2023-02-13 11:59:31,775 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165 2023-02-13 11:59:31,786 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-13 11:59:31,797 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 11:59:31,797 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23, deleteOnExit=true 2023-02-13 11:59:31,797 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 11:59:31,798 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/test.cache.data in system properties and HBase conf 2023-02-13 11:59:31,798 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 11:59:31,798 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/hadoop.log.dir in system properties and HBase conf 2023-02-13 11:59:31,799 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 11:59:31,799 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 11:59:31,799 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 11:59:31,888 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 11:59:32,193 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 11:59:32,197 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 11:59:32,197 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 11:59:32,197 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 11:59:32,198 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 11:59:32,198 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 11:59:32,198 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 11:59:32,198 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 11:59:32,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 11:59:32,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 11:59:32,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/nfs.dump.dir in system properties and HBase conf 2023-02-13 11:59:32,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/java.io.tmpdir in system properties and HBase conf 2023-02-13 11:59:32,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 11:59:32,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 11:59:32,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 11:59:32,614 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 11:59:32,617 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 11:59:33,053 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 11:59:33,186 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 11:59:33,200 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 11:59:33,231 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 11:59:33,260 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/java.io.tmpdir/Jetty_localhost_localdomain_43389_hdfs____nj60kv/webapp 2023-02-13 11:59:33,360 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:43389 2023-02-13 11:59:33,389 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 11:59:33,390 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 11:59:33,851 WARN [Listener at localhost.localdomain/41081] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 11:59:33,930 WARN [Listener at localhost.localdomain/41081] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 11:59:33,946 WARN [Listener at localhost.localdomain/41081] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 11:59:33,951 INFO [Listener at localhost.localdomain/41081] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 11:59:33,956 INFO [Listener at localhost.localdomain/41081] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/java.io.tmpdir/Jetty_localhost_46557_datanode____bdaa7w/webapp 2023-02-13 11:59:34,031 INFO [Listener at localhost.localdomain/41081] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46557 2023-02-13 11:59:34,279 WARN [Listener at localhost.localdomain/37111] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 11:59:34,297 WARN [Listener at localhost.localdomain/37111] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 11:59:34,300 WARN [Listener at localhost.localdomain/37111] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 11:59:34,302 INFO [Listener at localhost.localdomain/37111] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 11:59:34,308 INFO [Listener at localhost.localdomain/37111] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/java.io.tmpdir/Jetty_localhost_37325_datanode____.yj5kqv/webapp 2023-02-13 11:59:34,387 INFO [Listener at localhost.localdomain/37111] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37325 2023-02-13 11:59:34,395 WARN [Listener at localhost.localdomain/39987] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 11:59:34,408 WARN [Listener at localhost.localdomain/39987] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 11:59:34,411 WARN [Listener at localhost.localdomain/39987] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 11:59:34,414 INFO [Listener at localhost.localdomain/39987] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 11:59:34,418 INFO [Listener at localhost.localdomain/39987] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/java.io.tmpdir/Jetty_localhost_34087_datanode____.1t2nl7/webapp 2023-02-13 11:59:34,505 INFO [Listener at localhost.localdomain/39987] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34087 2023-02-13 11:59:34,514 WARN [Listener at localhost.localdomain/36409] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 11:59:35,481 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa89fca325f7b1b30: Processing first storage report for DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1 from datanode 14795f5c-1d04-4a18-8e7a-59414b03ca82 2023-02-13 11:59:35,482 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa89fca325f7b1b30: from storage DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1 node DatanodeRegistration(127.0.0.1:38207, datanodeUuid=14795f5c-1d04-4a18-8e7a-59414b03ca82, infoPort=45367, infoSecurePort=0, ipcPort=36409, storageInfo=lv=-57;cid=testClusterID;nsid=1266750951;c=1676289572673), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-13 11:59:35,482 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x298ada6d93ae17e2: Processing first storage report for DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f from datanode 27381562-f408-4782-a1fa-cd8daf0674dd 2023-02-13 11:59:35,482 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x298ada6d93ae17e2: from storage DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f node DatanodeRegistration(127.0.0.1:43795, datanodeUuid=27381562-f408-4782-a1fa-cd8daf0674dd, infoPort=42767, infoSecurePort=0, ipcPort=37111, storageInfo=lv=-57;cid=testClusterID;nsid=1266750951;c=1676289572673), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 11:59:35,483 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5aed046c658c14cd: Processing first storage report for DS-5c24f016-6269-482e-85dd-0d454c40b787 from datanode e81e25ff-5c5b-437b-b4c3-1df197948ce3 2023-02-13 11:59:35,483 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5aed046c658c14cd: from storage DS-5c24f016-6269-482e-85dd-0d454c40b787 node DatanodeRegistration(127.0.0.1:43995, datanodeUuid=e81e25ff-5c5b-437b-b4c3-1df197948ce3, infoPort=33995, infoSecurePort=0, ipcPort=39987, storageInfo=lv=-57;cid=testClusterID;nsid=1266750951;c=1676289572673), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 11:59:35,483 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa89fca325f7b1b30: Processing first storage report for DS-fb9daf6e-3a97-4339-b93e-893db3753d80 from datanode 14795f5c-1d04-4a18-8e7a-59414b03ca82 2023-02-13 11:59:35,483 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa89fca325f7b1b30: from storage DS-fb9daf6e-3a97-4339-b93e-893db3753d80 node DatanodeRegistration(127.0.0.1:38207, datanodeUuid=14795f5c-1d04-4a18-8e7a-59414b03ca82, infoPort=45367, infoSecurePort=0, ipcPort=36409, storageInfo=lv=-57;cid=testClusterID;nsid=1266750951;c=1676289572673), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 11:59:35,483 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x298ada6d93ae17e2: Processing first storage report for DS-abb6e6f8-1200-4a83-b8af-63c683c924f7 from datanode 27381562-f408-4782-a1fa-cd8daf0674dd 2023-02-13 11:59:35,483 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x298ada6d93ae17e2: from storage DS-abb6e6f8-1200-4a83-b8af-63c683c924f7 node DatanodeRegistration(127.0.0.1:43795, datanodeUuid=27381562-f408-4782-a1fa-cd8daf0674dd, infoPort=42767, infoSecurePort=0, ipcPort=37111, storageInfo=lv=-57;cid=testClusterID;nsid=1266750951;c=1676289572673), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 11:59:35,483 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5aed046c658c14cd: Processing first storage report for DS-382a5111-ee18-40ef-938e-58b3487f8dce from datanode e81e25ff-5c5b-437b-b4c3-1df197948ce3 2023-02-13 11:59:35,484 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5aed046c658c14cd: from storage DS-382a5111-ee18-40ef-938e-58b3487f8dce node DatanodeRegistration(127.0.0.1:43995, datanodeUuid=e81e25ff-5c5b-437b-b4c3-1df197948ce3, infoPort=33995, infoSecurePort=0, ipcPort=39987, storageInfo=lv=-57;cid=testClusterID;nsid=1266750951;c=1676289572673), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 11:59:35,545 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165 2023-02-13 11:59:35,597 INFO [Listener at localhost.localdomain/36409] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/zookeeper_0, clientPort=63172, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 11:59:35,609 INFO [Listener at localhost.localdomain/36409] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63172 2023-02-13 11:59:35,615 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:35,618 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:36,244 INFO [Listener at localhost.localdomain/36409] util.FSUtils(479): Created version file at hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2 with version=8 2023-02-13 11:59:36,244 INFO [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/hbase-staging 2023-02-13 11:59:36,483 INFO [Listener at localhost.localdomain/36409] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 11:59:36,869 INFO [Listener at localhost.localdomain/36409] client.ConnectionUtils(127): master/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 11:59:36,895 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:36,895 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:36,896 INFO [Listener at localhost.localdomain/36409] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 11:59:36,896 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:36,896 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 11:59:37,006 INFO [Listener at localhost.localdomain/36409] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 11:59:37,065 DEBUG [Listener at localhost.localdomain/36409] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 11:59:37,140 INFO [Listener at localhost.localdomain/36409] ipc.NettyRpcServer(120): Bind to /188.40.62.62:39205 2023-02-13 11:59:37,150 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,151 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,170 INFO [Listener at localhost.localdomain/36409] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39205 connecting to ZooKeeper ensemble=127.0.0.1:63172 2023-02-13 11:59:37,268 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:392050x0, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 11:59:37,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39205-0x10052c279140000 connected 2023-02-13 11:59:37,347 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 11:59:37,348 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 11:59:37,352 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 11:59:37,360 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39205 2023-02-13 11:59:37,360 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39205 2023-02-13 11:59:37,360 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39205 2023-02-13 11:59:37,361 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39205 2023-02-13 11:59:37,361 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39205 2023-02-13 11:59:37,366 INFO [Listener at localhost.localdomain/36409] master.HMaster(439): hbase.rootdir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2, hbase.cluster.distributed=false 2023-02-13 11:59:37,428 INFO [Listener at localhost.localdomain/36409] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 11:59:37,428 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,428 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,428 INFO [Listener at localhost.localdomain/36409] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 11:59:37,429 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,429 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 11:59:37,432 INFO [Listener at localhost.localdomain/36409] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 11:59:37,435 INFO [Listener at localhost.localdomain/36409] ipc.NettyRpcServer(120): Bind to /188.40.62.62:35305 2023-02-13 11:59:37,437 INFO [Listener at localhost.localdomain/36409] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 11:59:37,442 INFO [Listener at localhost.localdomain/36409] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 11:59:37,443 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,445 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,447 INFO [Listener at localhost.localdomain/36409] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35305 connecting to ZooKeeper ensemble=127.0.0.1:63172 2023-02-13 11:59:37,459 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:353050x0, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 11:59:37,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35305-0x10052c279140001 connected 2023-02-13 11:59:37,460 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 11:59:37,461 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 11:59:37,462 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 11:59:37,463 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35305 2023-02-13 11:59:37,463 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35305 2023-02-13 11:59:37,463 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35305 2023-02-13 11:59:37,464 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35305 2023-02-13 11:59:37,464 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35305 2023-02-13 11:59:37,479 INFO [Listener at localhost.localdomain/36409] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 11:59:37,479 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,479 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,480 INFO [Listener at localhost.localdomain/36409] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 11:59:37,480 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,481 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 11:59:37,481 INFO [Listener at localhost.localdomain/36409] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 11:59:37,482 INFO [Listener at localhost.localdomain/36409] ipc.NettyRpcServer(120): Bind to /188.40.62.62:45135 2023-02-13 11:59:37,483 INFO [Listener at localhost.localdomain/36409] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 11:59:37,484 INFO [Listener at localhost.localdomain/36409] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 11:59:37,485 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,487 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,489 INFO [Listener at localhost.localdomain/36409] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45135 connecting to ZooKeeper ensemble=127.0.0.1:63172 2023-02-13 11:59:37,500 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:451350x0, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 11:59:37,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:45135-0x10052c279140002 connected 2023-02-13 11:59:37,502 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 11:59:37,503 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 11:59:37,503 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 11:59:37,504 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45135 2023-02-13 11:59:37,505 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45135 2023-02-13 11:59:37,505 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45135 2023-02-13 11:59:37,505 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45135 2023-02-13 11:59:37,506 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45135 2023-02-13 11:59:37,519 INFO [Listener at localhost.localdomain/36409] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 11:59:37,520 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,520 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,520 INFO [Listener at localhost.localdomain/36409] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 11:59:37,520 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 11:59:37,520 INFO [Listener at localhost.localdomain/36409] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 11:59:37,520 INFO [Listener at localhost.localdomain/36409] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 11:59:37,522 INFO [Listener at localhost.localdomain/36409] ipc.NettyRpcServer(120): Bind to /188.40.62.62:34233 2023-02-13 11:59:37,522 INFO [Listener at localhost.localdomain/36409] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 11:59:37,523 INFO [Listener at localhost.localdomain/36409] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 11:59:37,524 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,525 INFO [Listener at localhost.localdomain/36409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,527 INFO [Listener at localhost.localdomain/36409] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34233 connecting to ZooKeeper ensemble=127.0.0.1:63172 2023-02-13 11:59:37,537 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:342330x0, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 11:59:37,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34233-0x10052c279140003 connected 2023-02-13 11:59:37,538 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 11:59:37,539 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 11:59:37,539 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ZKUtil(164): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 11:59:37,540 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34233 2023-02-13 11:59:37,540 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34233 2023-02-13 11:59:37,541 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34233 2023-02-13 11:59:37,541 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34233 2023-02-13 11:59:37,541 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34233 2023-02-13 11:59:37,543 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:37,563 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 11:59:37,565 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:37,592 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 11:59:37,592 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 11:59:37,592 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 11:59:37,592 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 11:59:37,592 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:37,594 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 11:59:37,595 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase16.apache.org,39205,1676289576353 from backup master directory 2023-02-13 11:59:37,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 11:59:37,609 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:37,609 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 11:59:37,610 WARN [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 11:59:37,610 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:37,614 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 11:59:37,616 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 11:59:37,707 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/hbase.id with ID: 5c9da710-d602-43be-a60c-3a31711e4468 2023-02-13 11:59:37,752 INFO [master/jenkins-hbase16:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 11:59:37,775 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:37,827 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x140f10ec to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:37,861 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@714efcb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:37,879 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 11:59:37,880 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 11:59:37,893 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 11:59:37,894 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 11:59:37,895 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 11:59:37,898 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 11:59:37,899 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 11:59:37,925 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store-tmp 2023-02-13 11:59:37,994 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:37,994 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 11:59:37,995 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 11:59:37,995 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 11:59:37,995 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 11:59:37,995 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 11:59:37,995 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 11:59:37,995 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676289577994Disabling compacts and flushes for region at 1676289577994Disabling writes for close at 1676289577995 (+1 ms)Writing region close event to WAL at 1676289577995Closed at 1676289577995 2023-02-13 11:59:37,997 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/WALs/jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:38,017 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C39205%2C1676289576353, suffix=, logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/WALs/jenkins-hbase16.apache.org,39205,1676289576353, archiveDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/oldWALs, maxLogs=10 2023-02-13 11:59:38,065 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK] 2023-02-13 11:59:38,065 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK] 2023-02-13 11:59:38,065 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK] 2023-02-13 11:59:38,073 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 11:59:38,138 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/WALs/jenkins-hbase16.apache.org,39205,1676289576353/jenkins-hbase16.apache.org%2C39205%2C1676289576353.1676289578025 2023-02-13 11:59:38,139 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK], DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK], DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK]] 2023-02-13 11:59:38,139 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 11:59:38,140 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:38,143 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 11:59:38,144 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 11:59:38,196 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 11:59:38,204 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 11:59:38,226 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 11:59:38,237 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:38,242 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 11:59:38,244 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 11:59:38,257 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 11:59:38,261 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:38,262 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58782109, jitterRate=-0.12407831847667694}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:38,262 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676289578167Initializing all the Stores at 1676289578169 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676289578170 (+1 ms)Cleaning up temporary data from old regions at 1676289578249 (+79 ms)Cleaning up detritus from prior splits at 1676289578250 (+1 ms)Region opened successfully at 1676289578262 (+12 ms) 2023-02-13 11:59:38,263 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 11:59:38,279 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 11:59:38,280 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 11:59:38,282 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 11:59:38,283 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 11:59:38,308 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-02-13 11:59:38,308 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 11:59:38,329 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 11:59:38,334 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 11:59:38,354 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 11:59:38,357 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 11:59:38,358 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 11:59:38,362 INFO [master/jenkins-hbase16:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 11:59:38,366 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 11:59:38,375 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:38,377 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 11:59:38,377 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 11:59:38,388 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 11:59:38,400 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 11:59:38,400 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 11:59:38,400 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 11:59:38,400 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 11:59:38,401 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:38,401 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase16.apache.org,39205,1676289576353, sessionid=0x10052c279140000, setting cluster-up flag (Was=false) 2023-02-13 11:59:38,428 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:38,453 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 11:59:38,454 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:38,475 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:38,503 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 11:59:38,505 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:38,507 WARN [master/jenkins-hbase16:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp 2023-02-13 11:59:38,545 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(952): ClusterId : 5c9da710-d602-43be-a60c-3a31711e4468 2023-02-13 11:59:38,546 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(952): ClusterId : 5c9da710-d602-43be-a60c-3a31711e4468 2023-02-13 11:59:38,545 INFO [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(952): ClusterId : 5c9da710-d602-43be-a60c-3a31711e4468 2023-02-13 11:59:38,550 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 11:59:38,550 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 11:59:38,550 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 11:59:38,571 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 11:59:38,571 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 11:59:38,571 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 11:59:38,572 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 11:59:38,571 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 11:59:38,572 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 11:59:38,587 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 11:59:38,587 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 11:59:38,587 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 11:59:38,589 DEBUG [RS:1;jenkins-hbase16:45135] zookeeper.ReadOnlyZKClient(139): Connect 0x2ae8888a to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:38,590 DEBUG [RS:0;jenkins-hbase16:35305] zookeeper.ReadOnlyZKClient(139): Connect 0x6aa55d4b to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:38,590 DEBUG [RS:2;jenkins-hbase16:34233] zookeeper.ReadOnlyZKClient(139): Connect 0x46789225 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:38,608 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 11:59:38,612 DEBUG [RS:1;jenkins-hbase16:45135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@242f73a7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:38,613 DEBUG [RS:2;jenkins-hbase16:34233] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48bf649a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:38,613 DEBUG [RS:1;jenkins-hbase16:45135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2010e3fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 11:59:38,613 DEBUG [RS:2;jenkins-hbase16:34233] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c80f955, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 11:59:38,614 DEBUG [RS:0;jenkins-hbase16:35305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6462ae6f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:38,615 DEBUG [RS:0;jenkins-hbase16:35305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@605d363b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 11:59:38,621 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 11:59:38,622 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 11:59:38,622 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 11:59:38,622 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 11:59:38,622 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase16:0, corePoolSize=10, maxPoolSize=10 2023-02-13 11:59:38,622 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,622 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 11:59:38,623 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,628 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676289608628 2023-02-13 11:59:38,631 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 11:59:38,634 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 11:59:38,634 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 11:59:38,640 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 11:59:38,641 DEBUG [RS:1;jenkins-hbase16:45135] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase16:45135 2023-02-13 11:59:38,642 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 11:59:38,644 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase16:34233 2023-02-13 11:59:38,645 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase16:35305 2023-02-13 11:59:38,648 INFO [RS:2;jenkins-hbase16:34233] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 11:59:38,648 INFO [RS:1;jenkins-hbase16:45135] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 11:59:38,651 INFO [RS:2;jenkins-hbase16:34233] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 11:59:38,651 INFO [RS:0;jenkins-hbase16:35305] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 11:59:38,651 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 11:59:38,651 INFO [RS:0;jenkins-hbase16:35305] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 11:59:38,652 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 11:59:38,651 INFO [RS:1;jenkins-hbase16:45135] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 11:59:38,652 DEBUG [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 11:59:38,653 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 11:59:38,654 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 11:59:38,655 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 11:59:38,655 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,39205,1676289576353 with isa=jenkins-hbase16.apache.org/188.40.62.62:34233, startcode=1676289577518 2023-02-13 11:59:38,655 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 11:59:38,655 INFO [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,39205,1676289576353 with isa=jenkins-hbase16.apache.org/188.40.62.62:45135, startcode=1676289577478 2023-02-13 11:59:38,655 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,39205,1676289576353 with isa=jenkins-hbase16.apache.org/188.40.62.62:35305, startcode=1676289577427 2023-02-13 11:59:38,656 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,657 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 11:59:38,659 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 11:59:38,659 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 11:59:38,665 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 11:59:38,666 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 11:59:38,667 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1676289578667,5,FailOnTimeoutGroup] 2023-02-13 11:59:38,668 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1676289578668,5,FailOnTimeoutGroup] 2023-02-13 11:59:38,668 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,668 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 11:59:38,670 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,670 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,675 DEBUG [RS:1;jenkins-hbase16:45135] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 11:59:38,675 DEBUG [RS:2;jenkins-hbase16:34233] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 11:59:38,675 DEBUG [RS:0;jenkins-hbase16:35305] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 11:59:38,698 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 11:59:38,699 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 11:59:38,699 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2 2023-02-13 11:59:38,735 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:38,739 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 11:59:38,742 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/info 2023-02-13 11:59:38,744 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 11:59:38,745 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:38,745 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 11:59:38,748 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 11:59:38,748 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 11:59:38,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:38,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 11:59:38,751 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/table 2023-02-13 11:59:38,752 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 11:59:38,752 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:59653, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 11:59:38,752 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:43391, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 11:59:38,752 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:47051, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 11:59:38,753 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:38,755 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740 2023-02-13 11:59:38,756 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740 2023-02-13 11:59:38,760 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 11:59:38,763 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 11:59:38,766 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39205] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39205] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,767 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,768 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:38,768 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61181653, jitterRate=-0.08832232654094696}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 11:59:38,768 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676289578736Initializing all the Stores at 1676289578738 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676289578738Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676289578739 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676289578739Cleaning up temporary data from old regions at 1676289578758 (+19 ms)Cleaning up detritus from prior splits at 1676289578758Region opened successfully at 1676289578768 (+10 ms) 2023-02-13 11:59:38,769 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 11:59:38,769 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 11:59:38,769 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 11:59:38,769 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 11:59:38,769 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 11:59:38,769 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 11:59:38,770 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676289578769Disabling compacts and flushes for region at 1676289578769Disabling writes for close at 1676289578769Writing region close event to WAL at 1676289578769Closed at 1676289578769 2023-02-13 11:59:38,778 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 11:59:38,778 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 11:59:38,786 DEBUG [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2 2023-02-13 11:59:38,787 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2 2023-02-13 11:59:38,787 DEBUG [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:41081 2023-02-13 11:59:38,787 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2 2023-02-13 11:59:38,787 DEBUG [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 11:59:38,787 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:41081 2023-02-13 11:59:38,787 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:41081 2023-02-13 11:59:38,787 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 11:59:38,787 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 11:59:38,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 11:59:38,800 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 11:59:38,802 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:38,830 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 11:59:38,845 DEBUG [RS:0;jenkins-hbase16:35305] zookeeper.ZKUtil(162): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,845 DEBUG [RS:2;jenkins-hbase16:34233] zookeeper.ZKUtil(162): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,845 WARN [RS:0;jenkins-hbase16:35305] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 11:59:38,845 WARN [RS:2;jenkins-hbase16:34233] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 11:59:38,847 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,45135,1676289577478] 2023-02-13 11:59:38,847 INFO [RS:2;jenkins-hbase16:34233] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 11:59:38,846 INFO [RS:0;jenkins-hbase16:35305] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 11:59:38,846 DEBUG [RS:1;jenkins-hbase16:45135] zookeeper.ZKUtil(162): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,848 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,848 WARN [RS:1;jenkins-hbase16:45135] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 11:59:38,848 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,847 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,35305,1676289577427] 2023-02-13 11:59:38,848 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,34233,1676289577518] 2023-02-13 11:59:38,848 INFO [RS:1;jenkins-hbase16:45135] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 11:59:38,849 DEBUG [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,865 DEBUG [RS:1;jenkins-hbase16:45135] zookeeper.ZKUtil(162): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,865 DEBUG [RS:2;jenkins-hbase16:34233] zookeeper.ZKUtil(162): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,865 DEBUG [RS:0;jenkins-hbase16:35305] zookeeper.ZKUtil(162): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,866 DEBUG [RS:1;jenkins-hbase16:45135] zookeeper.ZKUtil(162): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,866 DEBUG [RS:2;jenkins-hbase16:34233] zookeeper.ZKUtil(162): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,866 DEBUG [RS:0;jenkins-hbase16:35305] zookeeper.ZKUtil(162): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,867 DEBUG [RS:1;jenkins-hbase16:45135] zookeeper.ZKUtil(162): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,867 DEBUG [RS:0;jenkins-hbase16:35305] zookeeper.ZKUtil(162): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,867 DEBUG [RS:2;jenkins-hbase16:34233] zookeeper.ZKUtil(162): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,879 DEBUG [RS:1;jenkins-hbase16:45135] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 11:59:38,879 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 11:59:38,879 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 11:59:38,887 INFO [RS:1;jenkins-hbase16:45135] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 11:59:38,888 INFO [RS:2;jenkins-hbase16:34233] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 11:59:38,887 INFO [RS:0;jenkins-hbase16:35305] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 11:59:38,913 INFO [RS:0;jenkins-hbase16:35305] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 11:59:38,913 INFO [RS:2;jenkins-hbase16:34233] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 11:59:38,914 INFO [RS:1;jenkins-hbase16:45135] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 11:59:38,918 INFO [RS:2;jenkins-hbase16:34233] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 11:59:38,918 INFO [RS:1;jenkins-hbase16:45135] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 11:59:38,918 INFO [RS:2;jenkins-hbase16:34233] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,918 INFO [RS:0;jenkins-hbase16:35305] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 11:59:38,918 INFO [RS:1;jenkins-hbase16:45135] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,919 INFO [RS:0;jenkins-hbase16:35305] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,919 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 11:59:38,919 INFO [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 11:59:38,920 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 11:59:38,927 INFO [RS:2;jenkins-hbase16:34233] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,927 INFO [RS:1;jenkins-hbase16:45135] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,927 INFO [RS:0;jenkins-hbase16:35305] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,927 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,927 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,928 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 11:59:38,929 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 11:59:38,928 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:2;jenkins-hbase16:34233] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:0;jenkins-hbase16:35305] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,929 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,930 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,930 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 11:59:38,931 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,931 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,931 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,931 DEBUG [RS:1;jenkins-hbase16:45135] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 11:59:38,932 INFO [RS:2;jenkins-hbase16:34233] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,932 INFO [RS:2;jenkins-hbase16:34233] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,932 INFO [RS:0;jenkins-hbase16:35305] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,933 INFO [RS:2;jenkins-hbase16:34233] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,933 INFO [RS:0;jenkins-hbase16:35305] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,933 INFO [RS:0;jenkins-hbase16:35305] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,941 INFO [RS:1;jenkins-hbase16:45135] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,941 INFO [RS:1;jenkins-hbase16:45135] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,941 INFO [RS:1;jenkins-hbase16:45135] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,954 INFO [RS:2;jenkins-hbase16:34233] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 11:59:38,954 INFO [RS:0;jenkins-hbase16:35305] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 11:59:38,955 DEBUG [jenkins-hbase16:39205] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 11:59:38,955 INFO [RS:1;jenkins-hbase16:45135] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 11:59:38,957 INFO [RS:2;jenkins-hbase16:34233] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34233,1676289577518-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,957 INFO [RS:0;jenkins-hbase16:35305] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,35305,1676289577427-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,957 INFO [RS:1;jenkins-hbase16:45135] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,45135,1676289577478-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:38,959 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 11:59:38,967 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:59:38,967 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 11:59:38,967 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 11:59:38,967 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 11:59:38,970 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,35305,1676289577427, state=OPENING 2023-02-13 11:59:38,979 INFO [RS:2;jenkins-hbase16:34233] regionserver.Replication(203): jenkins-hbase16.apache.org,34233,1676289577518 started 2023-02-13 11:59:38,979 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,34233,1676289577518, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:34233, sessionid=0x10052c279140003 2023-02-13 11:59:38,980 INFO [RS:1;jenkins-hbase16:45135] regionserver.Replication(203): jenkins-hbase16.apache.org,45135,1676289577478 started 2023-02-13 11:59:38,980 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 11:59:38,980 INFO [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,45135,1676289577478, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:45135, sessionid=0x10052c279140002 2023-02-13 11:59:38,980 DEBUG [RS:2;jenkins-hbase16:34233] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,980 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 11:59:38,980 DEBUG [RS:2;jenkins-hbase16:34233] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34233,1676289577518' 2023-02-13 11:59:38,980 DEBUG [RS:1;jenkins-hbase16:45135] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,980 DEBUG [RS:2;jenkins-hbase16:34233] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 11:59:38,981 DEBUG [RS:1;jenkins-hbase16:45135] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,45135,1676289577478' 2023-02-13 11:59:38,981 DEBUG [RS:1;jenkins-hbase16:45135] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 11:59:38,984 INFO [RS:0;jenkins-hbase16:35305] regionserver.Replication(203): jenkins-hbase16.apache.org,35305,1676289577427 started 2023-02-13 11:59:38,985 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,35305,1676289577427, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:35305, sessionid=0x10052c279140001 2023-02-13 11:59:38,985 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 11:59:38,985 DEBUG [RS:0;jenkins-hbase16:35305] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,985 DEBUG [RS:0;jenkins-hbase16:35305] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,35305,1676289577427' 2023-02-13 11:59:38,985 DEBUG [RS:0;jenkins-hbase16:35305] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 11:59:38,986 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 11:59:38,986 DEBUG [RS:0;jenkins-hbase16:35305] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 11:59:38,986 DEBUG [RS:2;jenkins-hbase16:34233] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 11:59:38,988 DEBUG [RS:1;jenkins-hbase16:45135] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 11:59:38,995 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:38,995 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 11:59:38,995 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 11:59:38,995 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 11:59:38,995 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 11:59:38,995 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 11:59:38,995 DEBUG [RS:0;jenkins-hbase16:35305] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:38,995 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 11:59:38,996 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 11:59:38,996 DEBUG [RS:0;jenkins-hbase16:35305] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,35305,1676289577427' 2023-02-13 11:59:38,996 DEBUG [RS:0;jenkins-hbase16:35305] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 11:59:38,995 DEBUG [RS:2;jenkins-hbase16:34233] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:38,996 DEBUG [RS:1;jenkins-hbase16:45135] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:38,997 DEBUG [RS:2;jenkins-hbase16:34233] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34233,1676289577518' 2023-02-13 11:59:38,997 DEBUG [RS:2;jenkins-hbase16:34233] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 11:59:38,997 DEBUG [RS:1;jenkins-hbase16:45135] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,45135,1676289577478' 2023-02-13 11:59:38,997 DEBUG [RS:1;jenkins-hbase16:45135] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 11:59:38,997 DEBUG [RS:0;jenkins-hbase16:35305] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 11:59:38,997 DEBUG [RS:2;jenkins-hbase16:34233] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 11:59:38,997 DEBUG [RS:1;jenkins-hbase16:45135] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 11:59:38,997 DEBUG [RS:0;jenkins-hbase16:35305] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 11:59:38,998 DEBUG [RS:2;jenkins-hbase16:34233] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 11:59:38,998 INFO [RS:0;jenkins-hbase16:35305] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 11:59:38,998 INFO [RS:2;jenkins-hbase16:34233] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 11:59:38,998 INFO [RS:0;jenkins-hbase16:35305] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 11:59:38,998 INFO [RS:2;jenkins-hbase16:34233] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 11:59:38,999 DEBUG [RS:1;jenkins-hbase16:45135] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 11:59:38,999 INFO [RS:1;jenkins-hbase16:45135] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 11:59:38,999 INFO [RS:1;jenkins-hbase16:45135] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 11:59:39,001 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:39,114 INFO [RS:2;jenkins-hbase16:34233] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C34233%2C1676289577518, suffix=, logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,34233,1676289577518, archiveDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs, maxLogs=32 2023-02-13 11:59:39,114 INFO [RS:1;jenkins-hbase16:45135] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C45135%2C1676289577478, suffix=, logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,45135,1676289577478, archiveDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs, maxLogs=32 2023-02-13 11:59:39,114 INFO [RS:0;jenkins-hbase16:35305] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C35305%2C1676289577427, suffix=, logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,35305,1676289577427, archiveDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs, maxLogs=32 2023-02-13 11:59:39,139 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK] 2023-02-13 11:59:39,139 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK] 2023-02-13 11:59:39,139 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK] 2023-02-13 11:59:39,139 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK] 2023-02-13 11:59:39,139 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK] 2023-02-13 11:59:39,139 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK] 2023-02-13 11:59:39,140 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK] 2023-02-13 11:59:39,140 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK] 2023-02-13 11:59:39,140 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK] 2023-02-13 11:59:39,162 INFO [RS:2;jenkins-hbase16:34233] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,34233,1676289577518/jenkins-hbase16.apache.org%2C34233%2C1676289577518.1676289579117 2023-02-13 11:59:39,162 INFO [RS:1;jenkins-hbase16:45135] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,45135,1676289577478/jenkins-hbase16.apache.org%2C45135%2C1676289577478.1676289579117 2023-02-13 11:59:39,162 INFO [RS:0;jenkins-hbase16:35305] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,35305,1676289577427/jenkins-hbase16.apache.org%2C35305%2C1676289577427.1676289579117 2023-02-13 11:59:39,162 DEBUG [RS:2;jenkins-hbase16:34233] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK], DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK], DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK]] 2023-02-13 11:59:39,162 DEBUG [RS:1;jenkins-hbase16:45135] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK], DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK], DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK]] 2023-02-13 11:59:39,162 DEBUG [RS:0;jenkins-hbase16:35305] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK], DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK], DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK]] 2023-02-13 11:59:39,187 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:39,190 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 11:59:39,193 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37088, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 11:59:39,204 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 11:59:39,205 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 11:59:39,209 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C35305%2C1676289577427.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,35305,1676289577427, archiveDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs, maxLogs=32 2023-02-13 11:59:39,227 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK] 2023-02-13 11:59:39,227 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK] 2023-02-13 11:59:39,229 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK] 2023-02-13 11:59:39,236 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,35305,1676289577427/jenkins-hbase16.apache.org%2C35305%2C1676289577427.meta.1676289579210.meta 2023-02-13 11:59:39,236 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38207,DS-08a836fb-69b0-4e76-afdd-be4bbad90ab1,DISK], DatanodeInfoWithStorage[127.0.0.1:43995,DS-5c24f016-6269-482e-85dd-0d454c40b787,DISK], DatanodeInfoWithStorage[127.0.0.1:43795,DS-66d52e95-6ec9-44e5-8f27-d7e7736f2e6f,DISK]] 2023-02-13 11:59:39,236 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 11:59:39,238 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 11:59:39,251 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 11:59:39,256 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 11:59:39,260 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 11:59:39,261 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:39,261 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 11:59:39,261 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 11:59:39,263 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 11:59:39,265 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/info 2023-02-13 11:59:39,265 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/info 2023-02-13 11:59:39,266 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 11:59:39,267 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:39,267 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 11:59:39,268 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 11:59:39,268 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/rep_barrier 2023-02-13 11:59:39,269 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 11:59:39,270 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:39,270 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 11:59:39,271 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/table 2023-02-13 11:59:39,272 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/table 2023-02-13 11:59:39,272 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 11:59:39,273 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:39,276 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740 2023-02-13 11:59:39,279 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740 2023-02-13 11:59:39,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 11:59:39,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 11:59:39,288 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67833613, jitterRate=0.010799601674079895}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 11:59:39,288 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676289579261Writing region info on filesystem at 1676289579261Initializing all the Stores at 1676289579263 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676289579263Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676289579263Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676289579263Cleaning up temporary data from old regions at 1676289579280 (+17 ms)Cleaning up detritus from prior splits at 1676289579281 (+1 ms)Running coprocessor post-open hooks at 1676289579288 (+7 ms)Region opened successfully at 1676289579288 2023-02-13 11:59:39,297 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676289579178 2023-02-13 11:59:39,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 11:59:39,315 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 11:59:39,315 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,35305,1676289577427, state=OPEN 2023-02-13 11:59:39,325 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 11:59:39,325 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 11:59:39,330 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 11:59:39,330 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,35305,1676289577427 in 324 msec 2023-02-13 11:59:39,337 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 11:59:39,337 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 543 msec 2023-02-13 11:59:39,343 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 795 msec 2023-02-13 11:59:39,344 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676289577580, completionTime=-1 2023-02-13 11:59:39,344 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 11:59:39,344 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 11:59:39,399 DEBUG [hconnection-0x1b9902f2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:39,402 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37092, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:39,417 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 11:59:39,417 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676289639417 2023-02-13 11:59:39,417 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676289699417 2023-02-13 11:59:39,417 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 73 msec 2023-02-13 11:59:39,472 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,39205,1676289576353-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:39,472 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,39205,1676289576353-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:39,473 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,39205,1676289576353-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:39,474 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase16:39205, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:39,474 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 11:59:39,484 DEBUG [master/jenkins-hbase16:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 11:59:39,490 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 11:59:39,491 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 11:59:39,505 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 11:59:39,513 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 11:59:39,516 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 11:59:39,537 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:39,540 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f empty. 2023-02-13 11:59:39,541 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:39,541 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 11:59:39,581 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 11:59:39,583 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => df3c86cd8ebebefe70d222a681f7de3f, NAME => 'hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 11:59:39,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:39,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing df3c86cd8ebebefe70d222a681f7de3f, disabling compactions & flushes 2023-02-13 11:59:39,605 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 11:59:39,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 11:59:39,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. after waiting 0 ms 2023-02-13 11:59:39,605 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 11:59:39,606 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 11:59:39,606 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for df3c86cd8ebebefe70d222a681f7de3f: Waiting for close lock at 1676289579605Disabling compacts and flushes for region at 1676289579605Disabling writes for close at 1676289579605Writing region close event to WAL at 1676289579606 (+1 ms)Closed at 1676289579606 2023-02-13 11:59:39,611 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 11:59:39,625 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676289579613"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289579613"}]},"ts":"1676289579613"} 2023-02-13 11:59:39,652 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 11:59:39,654 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 11:59:39,694 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289579654"}]},"ts":"1676289579654"} 2023-02-13 11:59:39,700 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 11:59:39,720 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 11:59:39,722 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:59:39,722 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 11:59:39,722 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 11:59:39,722 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 11:59:39,724 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=df3c86cd8ebebefe70d222a681f7de3f, ASSIGN}] 2023-02-13 11:59:39,728 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=df3c86cd8ebebefe70d222a681f7de3f, ASSIGN 2023-02-13 11:59:39,730 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=df3c86cd8ebebefe70d222a681f7de3f, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=false 2023-02-13 11:59:39,882 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 11:59:39,883 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=df3c86cd8ebebefe70d222a681f7de3f, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:39,884 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676289579883"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289579883"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289579883"}]},"ts":"1676289579883"} 2023-02-13 11:59:39,889 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure df3c86cd8ebebefe70d222a681f7de3f, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:40,043 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:40,044 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 11:59:40,048 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48664, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 11:59:40,057 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 11:59:40,061 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => df3c86cd8ebebefe70d222a681f7de3f, NAME => 'hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.', STARTKEY => '', ENDKEY => ''} 2023-02-13 11:59:40,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:40,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:40,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:40,062 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:40,066 INFO [StoreOpener-df3c86cd8ebebefe70d222a681f7de3f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:40,069 DEBUG [StoreOpener-df3c86cd8ebebefe70d222a681f7de3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/info 2023-02-13 11:59:40,069 DEBUG [StoreOpener-df3c86cd8ebebefe70d222a681f7de3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/info 2023-02-13 11:59:40,070 INFO [StoreOpener-df3c86cd8ebebefe70d222a681f7de3f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region df3c86cd8ebebefe70d222a681f7de3f columnFamilyName info 2023-02-13 11:59:40,071 INFO [StoreOpener-df3c86cd8ebebefe70d222a681f7de3f-1] regionserver.HStore(310): Store=df3c86cd8ebebefe70d222a681f7de3f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:40,073 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:40,074 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:40,079 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 11:59:40,083 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:40,084 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened df3c86cd8ebebefe70d222a681f7de3f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74119156, jitterRate=0.10446149110794067}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:40,085 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for df3c86cd8ebebefe70d222a681f7de3f: Running coprocessor pre-open hook at 1676289580063Writing region info on filesystem at 1676289580063Initializing all the Stores at 1676289580064 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676289580064Cleaning up temporary data from old regions at 1676289580075 (+11 ms)Cleaning up detritus from prior splits at 1676289580076 (+1 ms)Running coprocessor post-open hooks at 1676289580085 (+9 ms)Region opened successfully at 1676289580085 2023-02-13 11:59:40,087 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f., pid=6, masterSystemTime=1676289580043 2023-02-13 11:59:40,092 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 11:59:40,093 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 11:59:40,095 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=df3c86cd8ebebefe70d222a681f7de3f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:40,096 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676289580094"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289580094"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289580094"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289580094"}]},"ts":"1676289580094"} 2023-02-13 11:59:40,104 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 11:59:40,104 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure df3c86cd8ebebefe70d222a681f7de3f, server=jenkins-hbase16.apache.org,34233,1676289577518 in 210 msec 2023-02-13 11:59:40,108 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 11:59:40,109 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=df3c86cd8ebebefe70d222a681f7de3f, ASSIGN in 380 msec 2023-02-13 11:59:40,110 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 11:59:40,110 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289580110"}]},"ts":"1676289580110"} 2023-02-13 11:59:40,113 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 11:59:40,163 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 11:59:40,166 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 670 msec 2023-02-13 11:59:40,209 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 11:59:40,242 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 11:59:40,242 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 11:59:40,262 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:40,264 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48670, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:40,280 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 11:59:40,303 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 11:59:40,319 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 47 msec 2023-02-13 11:59:40,323 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 11:59:40,345 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 11:59:40,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 33 msec 2023-02-13 11:59:40,383 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 11:59:40,400 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 11:59:40,400 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.790sec 2023-02-13 11:59:40,403 INFO [master/jenkins-hbase16:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 11:59:40,404 INFO [master/jenkins-hbase16:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 11:59:40,404 INFO [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 11:59:40,405 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,39205,1676289576353-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 11:59:40,407 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,39205,1676289576353-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 11:59:40,415 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 11:59:40,454 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(139): Connect 0x76012f12 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:40,468 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cffe5a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:40,482 DEBUG [hconnection-0x6136c840-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:40,493 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52026, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:40,502 INFO [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 11:59:40,532 INFO [Listener at localhost.localdomain/36409] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=363, OpenFileDescriptor=697, MaxFileDescriptor=60000, SystemLoadAverage=280, ProcessCount=171, AvailableMemoryMB=6109 2023-02-13 11:59:40,553 DEBUG [Listener at localhost.localdomain/36409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 11:59:40,560 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52694, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 11:59:40,575 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$4(2102): Client=jenkins//188.40.62.62 create 'testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 11:59:40,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:40,581 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676289580536 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 11:59:40,584 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676289580536 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 11:59:40,586 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676289580536" procId is: 9 2023-02-13 11:59:40,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 11:59:40,594 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:40,594 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:40,594 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:40,595 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:40,595 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:40,595 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf empty. 2023-02-13 11:59:40,596 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 empty. 2023-02-13 11:59:40,596 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:40,596 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d empty. 2023-02-13 11:59:40,596 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c empty. 2023-02-13 11:59:40,599 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:40,599 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:40,600 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:40,601 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a empty. 2023-02-13 11:59:40,601 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 empty. 2023-02-13 11:59:40,601 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:40,602 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:40,602 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:40,602 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676289580536 regions 2023-02-13 11:59:40,634 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000001 2023-02-13 11:59:40,636 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(7671): creating {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 11:59:40,637 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(7671): creating {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 11:59:40,640 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(7671): creating {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 11:59:40,702 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:40,705 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 11:59:40,705 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:40,705 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:40,705 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 11:59:40,705 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:40,705 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:40,706 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 11:59:40,705 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:40,706 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:40,706 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289580705Disabling compacts and flushes for region at 1676289580705Disabling writes for close at 1676289580705Writing region close event to WAL at 1676289580705Closed at 1676289580705 2023-02-13 11:59:40,706 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:40,707 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 11:59:40,707 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:40,707 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:40,707 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289580706Disabling compacts and flushes for region at 1676289580706Disabling writes for close at 1676289580707 (+1 ms)Writing region close event to WAL at 1676289580707Closed at 1676289580707 2023-02-13 11:59:40,708 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(7671): creating {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 11:59:40,709 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(7671): creating {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 11:59:40,711 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:40,712 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 11:59:40,712 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:40,712 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:40,712 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 11:59:40,712 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:40,712 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:40,712 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289580712Disabling compacts and flushes for region at 1676289580712Disabling writes for close at 1676289580712Writing region close event to WAL at 1676289580712Closed at 1676289580712 2023-02-13 11:59:40,713 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(7671): creating {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 11:59:40,749 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:40,753 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 11:59:40,753 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:40,753 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:40,753 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 11:59:40,753 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:40,753 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:40,753 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289580753Disabling compacts and flushes for region at 1676289580753Disabling writes for close at 1676289580753Writing region close event to WAL at 1676289580753Closed at 1676289580753 2023-02-13 11:59:40,755 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:40,755 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 11:59:40,756 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:40,756 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:40,756 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 11:59:40,756 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:40,756 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:40,756 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-1] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289580755Disabling compacts and flushes for region at 1676289580755Disabling writes for close at 1676289580756 (+1 ms)Writing region close event to WAL at 1676289580756Closed at 1676289580756 2023-02-13 11:59:40,761 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:40,761 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 11:59:40,761 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:40,761 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:40,761 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 11:59:40,761 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:40,761 INFO [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:40,761 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676289580536-pool-2] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289580761Disabling compacts and flushes for region at 1676289580761Disabling writes for close at 1676289580761Writing region close event to WAL at 1676289580761Closed at 1676289580761 2023-02-13 11:59:40,768 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676289580536 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 11:59:40,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289580769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289580769"}]},"ts":"1676289580769"} 2023-02-13 11:59:40,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289580769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289580769"}]},"ts":"1676289580769"} 2023-02-13 11:59:40,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289580769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289580769"}]},"ts":"1676289580769"} 2023-02-13 11:59:40,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289580769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289580769"}]},"ts":"1676289580769"} 2023-02-13 11:59:40,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289580769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289580769"}]},"ts":"1676289580769"} 2023-02-13 11:59:40,771 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289580769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289580769"}]},"ts":"1676289580769"} 2023-02-13 11:59:40,814 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 11:59:40,816 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676289580536 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 11:59:40,816 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289580816"}]},"ts":"1676289580816"} 2023-02-13 11:59:40,819 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLING in hbase:meta 2023-02-13 11:59:40,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 11:59:40,868 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 11:59:40,873 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:59:40,873 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 11:59:40,873 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 11:59:40,874 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 11:59:40,874 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN}] 2023-02-13 11:59:40,879 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN 2023-02-13 11:59:40,879 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN 2023-02-13 11:59:40,880 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN 2023-02-13 11:59:40,881 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN 2023-02-13 11:59:40,885 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN 2023-02-13 11:59:40,885 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=false 2023-02-13 11:59:40,885 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=false 2023-02-13 11:59:40,885 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=false 2023-02-13 11:59:40,885 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=false 2023-02-13 11:59:40,887 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=false 2023-02-13 11:59:40,887 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN 2023-02-13 11:59:40,889 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=false 2023-02-13 11:59:41,035 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 11:59:41,039 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:41,039 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:41,039 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:41,039 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:41,039 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:41,039 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581039"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581039"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581039"}]},"ts":"1676289581039"} 2023-02-13 11:59:41,039 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581039"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581039"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581039"}]},"ts":"1676289581039"} 2023-02-13 11:59:41,040 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581039"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581039"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581039"}]},"ts":"1676289581039"} 2023-02-13 11:59:41,039 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581039"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581039"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581039"}]},"ts":"1676289581039"} 2023-02-13 11:59:41,039 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581039"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581039"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581039"}]},"ts":"1676289581039"} 2023-02-13 11:59:41,042 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:41,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=11, state=RUNNABLE; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:41,047 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=14, state=RUNNABLE; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:41,050 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=13, state=RUNNABLE; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:41,051 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:41,052 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581051"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581051"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581051"}]},"ts":"1676289581051"} 2023-02-13 11:59:41,053 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=10, state=RUNNABLE; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:41,056 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=12, state=RUNNABLE; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:41,200 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:41,201 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 11:59:41,206 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57648, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 11:59:41,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:41,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 11:59:41,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:41,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:41,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 11:59:41,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:41,215 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,217 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:41,217 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:41,218 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,220 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,220 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b55b9ce69d00ead52d760ba43038e9a columnFamilyName cf 2023-02-13 11:59:41,221 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:41,221 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:41,222 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:41,222 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81f9e5dd49cc8e0565489f1ebf657e0d columnFamilyName cf 2023-02-13 11:59:41,222 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:41,223 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 56eee98ead755df3cb16e7153d171fbf columnFamilyName cf 2023-02-13 11:59:41,224 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(310): Store=2b55b9ce69d00ead52d760ba43038e9a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:41,224 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(310): Store=81f9e5dd49cc8e0565489f1ebf657e0d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:41,229 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(310): Store=56eee98ead755df3cb16e7153d171fbf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:41,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:41,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:41,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:41,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81f9e5dd49cc8e0565489f1ebf657e0d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70902683, jitterRate=0.05653230845928192}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:41,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Running coprocessor pre-open hook at 1676289581216Writing region info on filesystem at 1676289581216Initializing all the Stores at 1676289581217 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289581217Cleaning up temporary data from old regions at 1676289581232 (+15 ms)Cleaning up detritus from prior splits at 1676289581233 (+1 ms)Running coprocessor post-open hooks at 1676289581247 (+14 ms)Region opened successfully at 1676289581247 2023-02-13 11:59:41,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 56eee98ead755df3cb16e7153d171fbf; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60452168, jitterRate=-0.09919250011444092}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:41,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 2b55b9ce69d00ead52d760ba43038e9a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60364777, jitterRate=-0.10049472749233246}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:41,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 56eee98ead755df3cb16e7153d171fbf: Running coprocessor pre-open hook at 1676289581216Writing region info on filesystem at 1676289581216Initializing all the Stores at 1676289581217 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289581217Cleaning up temporary data from old regions at 1676289581233 (+16 ms)Cleaning up detritus from prior splits at 1676289581233Running coprocessor post-open hooks at 1676289581247 (+14 ms)Region opened successfully at 1676289581247 2023-02-13 11:59:41,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 2b55b9ce69d00ead52d760ba43038e9a: Running coprocessor pre-open hook at 1676289581210Writing region info on filesystem at 1676289581210Initializing all the Stores at 1676289581212 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289581212Cleaning up temporary data from old regions at 1676289581233 (+21 ms)Cleaning up detritus from prior splits at 1676289581234 (+1 ms)Running coprocessor post-open hooks at 1676289581248 (+14 ms)Region opened successfully at 1676289581248 2023-02-13 11:59:41,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., pid=20, masterSystemTime=1676289581206 2023-02-13 11:59:41,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., pid=17, masterSystemTime=1676289581200 2023-02-13 11:59:41,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., pid=16, masterSystemTime=1676289581198 2023-02-13 11:59:41,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:41,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 11:59:41,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,255 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:41,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:41,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,256 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581255"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289581255"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289581255"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289581255"}]},"ts":"1676289581255"} 2023-02-13 11:59:41,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:41,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:41,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:41,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 11:59:41,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:41,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,259 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,260 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:41,260 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581259"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289581259"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289581259"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289581259"}]},"ts":"1676289581259"} 2023-02-13 11:59:41,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:41,261 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:41,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:41,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 11:59:41,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:41,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,264 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:41,264 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:41,265 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a40bfc382ab282660f54577d5b7b9575 columnFamilyName cf 2023-02-13 11:59:41,265 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:41,265 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581265"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289581265"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289581265"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289581265"}]},"ts":"1676289581265"} 2023-02-13 11:59:41,266 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:41,267 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:41,268 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ee51e8905082f3fc07662881dfd80c columnFamilyName cf 2023-02-13 11:59:41,266 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(310): Store=a40bfc382ab282660f54577d5b7b9575/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:41,267 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,272 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(310): Store=e0ee51e8905082f3fc07662881dfd80c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:41,273 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-13 11:59:41,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,273 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 222 msec 2023-02-13 11:59:41,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=10 2023-02-13 11:59:41,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=10, state=SUCCESS; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 209 msec 2023-02-13 11:59:41,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,276 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:41,276 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:41,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,276 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c62ea170afdbf159d444aa319925e012 columnFamilyName cf 2023-02-13 11:59:41,277 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=11 2023-02-13 11:59:41,277 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN in 399 msec 2023-02-13 11:59:41,277 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=11, state=SUCCESS; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 227 msec 2023-02-13 11:59:41,277 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN in 399 msec 2023-02-13 11:59:41,279 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(310): Store=c62ea170afdbf159d444aa319925e012/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:41,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN in 403 msec 2023-02-13 11:59:41,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:41,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:41,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a40bfc382ab282660f54577d5b7b9575; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58997654, jitterRate=-0.12086644768714905}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:41,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a40bfc382ab282660f54577d5b7b9575: Running coprocessor pre-open hook at 1676289581256Writing region info on filesystem at 1676289581256Initializing all the Stores at 1676289581259 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289581259Cleaning up temporary data from old regions at 1676289581275 (+16 ms)Cleaning up detritus from prior splits at 1676289581276 (+1 ms)Running coprocessor post-open hooks at 1676289581292 (+16 ms)Region opened successfully at 1676289581292 2023-02-13 11:59:41,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened e0ee51e8905082f3fc07662881dfd80c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66751546, jitterRate=-0.005324453115463257}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:41,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for e0ee51e8905082f3fc07662881dfd80c: Running coprocessor pre-open hook at 1676289581257Writing region info on filesystem at 1676289581257Initializing all the Stores at 1676289581260 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289581260Cleaning up temporary data from old regions at 1676289581276 (+16 ms)Cleaning up detritus from prior splits at 1676289581278 (+2 ms)Running coprocessor post-open hooks at 1676289581293 (+15 ms)Region opened successfully at 1676289581293 2023-02-13 11:59:41,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., pid=19, masterSystemTime=1676289581206 2023-02-13 11:59:41,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., pid=21, masterSystemTime=1676289581198 2023-02-13 11:59:41,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:41,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:41,299 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:41,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:41,300 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581299"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289581299"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289581299"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289581299"}]},"ts":"1676289581299"} 2023-02-13 11:59:41,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 11:59:41,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:41,303 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:41,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened c62ea170afdbf159d444aa319925e012; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70128283, jitterRate=0.044992849230766296}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:41,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for c62ea170afdbf159d444aa319925e012: Running coprocessor pre-open hook at 1676289581264Writing region info on filesystem at 1676289581264Initializing all the Stores at 1676289581265 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289581266 (+1 ms)Cleaning up temporary data from old regions at 1676289581282 (+16 ms)Cleaning up detritus from prior splits at 1676289581290 (+8 ms)Running coprocessor post-open hooks at 1676289581303 (+13 ms)Region opened successfully at 1676289581303 2023-02-13 11:59:41,303 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581302"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289581302"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289581302"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289581302"}]},"ts":"1676289581302"} 2023-02-13 11:59:41,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., pid=18, masterSystemTime=1676289581200 2023-02-13 11:59:41,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:41,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:41,308 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:41,309 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581308"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289581308"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289581308"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289581308"}]},"ts":"1676289581308"} 2023-02-13 11:59:41,309 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=13 2023-02-13 11:59:41,309 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=13, state=SUCCESS; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 254 msec 2023-02-13 11:59:41,311 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=12 2023-02-13 11:59:41,311 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=12, state=SUCCESS; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 251 msec 2023-02-13 11:59:41,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN in 435 msec 2023-02-13 11:59:41,313 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN in 437 msec 2023-02-13 11:59:41,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=14 2023-02-13 11:59:41,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=14, state=SUCCESS; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 265 msec 2023-02-13 11:59:41,317 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=9 2023-02-13 11:59:41,317 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN in 440 msec 2023-02-13 11:59:41,318 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676289580536 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 11:59:41,319 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289581318"}]},"ts":"1676289581318"} 2023-02-13 11:59:41,321 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLED in hbase:meta 2023-02-13 11:59:41,338 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676289580536 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 11:59:41,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676289580536 in 763 msec 2023-02-13 11:59:41,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 11:59:41,354 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 9 completed 2023-02-13 11:59:41,361 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676289580536 2023-02-13 11:59:41,362 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,365 INFO [Listener at localhost.localdomain/36409] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 11:59:41,393 DEBUG [hconnection-0x6136c840-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:41,398 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57654, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:41,400 DEBUG [hconnection-0x6136c840-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:41,403 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48686, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:41,419 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:41,424 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676289580536 2023-02-13 11:59:41,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:41,434 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289581434"}]},"ts":"1676289581434"} 2023-02-13 11:59:41,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 11:59:41,436 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLING in hbase:meta 2023-02-13 11:59:41,533 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676289580536 to state=DISABLING 2023-02-13 11:59:41,536 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN}] 2023-02-13 11:59:41,539 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN 2023-02-13 11:59:41,539 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN 2023-02-13 11:59:41,539 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN 2023-02-13 11:59:41,540 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN 2023-02-13 11:59:41,540 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN 2023-02-13 11:59:41,541 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:41,541 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:41,541 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:41,541 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:41,542 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:41,542 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581541"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581541"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581541"}]},"ts":"1676289581541"} 2023-02-13 11:59:41,542 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581541"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581541"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581541"}]},"ts":"1676289581541"} 2023-02-13 11:59:41,542 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581541"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581541"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581541"}]},"ts":"1676289581541"} 2023-02-13 11:59:41,542 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581541"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581541"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581541"}]},"ts":"1676289581541"} 2023-02-13 11:59:41,542 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581542"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581542"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581542"}]},"ts":"1676289581542"} 2023-02-13 11:59:41,546 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:41,547 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=27, state=RUNNABLE; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:41,549 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=25, state=RUNNABLE; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:41,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=24, state=RUNNABLE; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:41,555 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:41,559 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN 2023-02-13 11:59:41,560 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:41,561 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581560"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289581560"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289581560"}]},"ts":"1676289581560"} 2023-02-13 11:59:41,565 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=26, state=RUNNABLE; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:41,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 11:59:41,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 11:59:41,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 11:59:41,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 11:59:41,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 11:59:41,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:41,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:41,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 11:59:41,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:41,733 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289581713Running coprocessor pre-close hooks at 1676289581713Disabling compacts and flushes for region at 1676289581713Disabling writes for close at 1676289581714 (+1 ms)Writing region close event to WAL at 1676289581722 (+8 ms)Running coprocessor post-close hooks at 1676289581733 (+11 ms)Closed at 1676289581733 2023-02-13 11:59:41,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:41,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,738 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSED 2023-02-13 11:59:41,739 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581738"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289581738"}]},"ts":"1676289581738"} 2023-02-13 11:59:41,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=27 2023-02-13 11:59:41,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=27, state=SUCCESS; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 196 msec 2023-02-13 11:59:41,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 11:59:41,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:41,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:41,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 11:59:41,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:41,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 11:59:41,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 11:59:41,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:41,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289581712Running coprocessor pre-close hooks at 1676289581712Disabling compacts and flushes for region at 1676289581712Disabling writes for close at 1676289581714 (+2 ms)Writing region close event to WAL at 1676289581723 (+9 ms)Running coprocessor post-close hooks at 1676289581752 (+29 ms)Closed at 1676289581752 2023-02-13 11:59:41,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:41,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289581712Running coprocessor pre-close hooks at 1676289581712Disabling compacts and flushes for region at 1676289581712Disabling writes for close at 1676289581714 (+2 ms)Writing region close event to WAL at 1676289581722 (+8 ms)Running coprocessor post-close hooks at 1676289581753 (+31 ms)Closed at 1676289581754 (+1 ms) 2023-02-13 11:59:41,754 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN in 211 msec 2023-02-13 11:59:41,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:41,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 11:59:41,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 11:59:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:41,760 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSED 2023-02-13 11:59:41,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:41,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 11:59:41,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:41,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:41,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 11:59:41,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:41,766 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289581760"}]},"ts":"1676289581760"} 2023-02-13 11:59:41,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 11:59:41,769 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSED 2023-02-13 11:59:41,769 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289581768"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289581768"}]},"ts":"1676289581768"} 2023-02-13 11:59:41,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:41,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289581737Running coprocessor pre-close hooks at 1676289581737Disabling compacts and flushes for region at 1676289581737Disabling writes for close at 1676289581749 (+12 ms)Writing region close event to WAL at 1676289581750 (+1 ms)Running coprocessor post-close hooks at 1676289581770 (+20 ms)Closed at 1676289581770 2023-02-13 11:59:41,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:41,774 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-13 11:59:41,775 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSED 2023-02-13 11:59:41,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 214 msec 2023-02-13 11:59:41,775 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581774"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289581774"}]},"ts":"1676289581774"} 2023-02-13 11:59:41,777 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-13 11:59:41,777 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 225 msec 2023-02-13 11:59:41,779 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN in 239 msec 2023-02-13 11:59:41,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN in 242 msec 2023-02-13 11:59:41,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 11:59:41,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 11:59:41,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:41,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289581756Running coprocessor pre-close hooks at 1676289581756Disabling compacts and flushes for region at 1676289581756Disabling writes for close at 1676289581757 (+1 ms)Writing region close event to WAL at 1676289581757Running coprocessor post-close hooks at 1676289581782 (+25 ms)Closed at 1676289581782 2023-02-13 11:59:41,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=24 2023-02-13 11:59:41,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:41,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=24, state=SUCCESS; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 225 msec 2023-02-13 11:59:41,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289581761Running coprocessor pre-close hooks at 1676289581761Disabling compacts and flushes for region at 1676289581761Disabling writes for close at 1676289581761Writing region close event to WAL at 1676289581766 (+5 ms)Running coprocessor post-close hooks at 1676289581782 (+16 ms)Closed at 1676289581782 2023-02-13 11:59:41,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:41,785 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN in 246 msec 2023-02-13 11:59:41,785 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSED 2023-02-13 11:59:41,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:41,785 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581785"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289581785"}]},"ts":"1676289581785"} 2023-02-13 11:59:41,786 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSED 2023-02-13 11:59:41,786 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289581786"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289581786"}]},"ts":"1676289581786"} 2023-02-13 11:59:41,791 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=25 2023-02-13 11:59:41,791 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=25, state=SUCCESS; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 238 msec 2023-02-13 11:59:41,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=26 2023-02-13 11:59:41,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=26, state=SUCCESS; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 224 msec 2023-02-13 11:59:41,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN in 255 msec 2023-02-13 11:59:41,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=22 2023-02-13 11:59:41,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN in 256 msec 2023-02-13 11:59:41,796 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289581796"}]},"ts":"1676289581796"} 2023-02-13 11:59:41,798 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLED in hbase:meta 2023-02-13 11:59:41,811 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676289580536 to state=DISABLED 2023-02-13 11:59:41,815 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676289580536 in 387 msec 2023-02-13 11:59:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 11:59:42,189 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 22 completed 2023-02-13 11:59:42,198 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:42,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289582198 (current time:1676289582198). 2023-02-13 11:59:42,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 11:59:42,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x546f3395 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:42,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cb5408b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:42,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:42,224 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52042, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:42,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x546f3395 to 127.0.0.1:63172 2023-02-13 11:59:42,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 11:59:42,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:42,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:42,243 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:42,245 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE; timeout=600000, timestamp=1676290182245 2023-02-13 11:59:42,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:42,245 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676289580536 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 2023-02-13 11:59:42,250 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 11:59:42,250 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 11:59:42,267 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 11:59:42,272 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:42,272 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:42,273 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:42,277 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:42,277 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:42,277 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:42,278 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:42,278 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:42,279 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:42,280 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:42,280 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:42,280 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:42,281 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:42,282 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:42,286 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 11:59:42,286 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 11:59:42,286 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 11:59:42,286 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 11:59:42,286 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 11:59:42,287 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 11:59:42,287 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 11:59:42,500 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 11:59:42,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 11:59:42,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 11:59:42,506 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 11:59:42,834 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } as finished. 2023-02-13 11:59:42,835 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676289580536 2023-02-13 11:59:42,838 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289580536 2023-02-13 11:59:42,952 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289580536 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/emptySnaptb-1676289580536 2023-02-13 11:59:43,001 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676289580536 of table testRestoreSnapshot-1676289580536 completed 2023-02-13 11:59:43,001 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289580536 2023-02-13 11:59:43,002 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289580536 2023-02-13 11:59:43,002 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676289580536 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 at 1676289582245Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } at 1676289582267 (+22 ms)Consolidate snapshot: emptySnaptb-1676289580536 at 1676289582835 (+568 ms)Loading Region manifests for emptySnaptb-1676289580536 at 1676289582835Writing data manifest for emptySnaptb-1676289580536 at 1676289582862 (+27 ms)Verifying snapshot: emptySnaptb-1676289580536 at 1676289582894 (+32 ms)Snapshot emptySnaptb-1676289580536 of table testRestoreSnapshot-1676289580536 completed at 1676289583001 (+107 ms) 2023-02-13 11:59:43,003 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:43,006 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 11:59:43,006 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE in 766 msec 2023-02-13 11:59:43,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 11:59:43,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 11:59:43,009 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,015 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 11:59:43,028 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,029 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289583029"}]},"ts":"1676289583029"} 2023-02-13 11:59:43,031 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLING in hbase:meta 2023-02-13 11:59:43,069 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 11:59:43,072 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN}] 2023-02-13 11:59:43,076 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN 2023-02-13 11:59:43,076 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN 2023-02-13 11:59:43,077 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN 2023-02-13 11:59:43,079 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN 2023-02-13 11:59:43,083 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,083 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,083 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 11:59:43,083 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,083 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 11:59:43,083 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 11:59:43,083 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN 2023-02-13 11:59:43,086 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,086 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 11:59:43,089 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,089 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 11:59:43,089 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN 2023-02-13 11:59:43,091 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,091 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 11:59:43,233 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 11:59:43,234 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,234 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,234 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289583234"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583234"}]},"ts":"1676289583234"} 2023-02-13 11:59:43,234 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289583234"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583234"}]},"ts":"1676289583234"} 2023-02-13 11:59:43,234 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,234 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,234 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583234"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583234"}]},"ts":"1676289583234"} 2023-02-13 11:59:43,234 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583234"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583234"}]},"ts":"1676289583234"} 2023-02-13 11:59:43,234 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,235 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583234"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583234"}]},"ts":"1676289583234"} 2023-02-13 11:59:43,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=37, state=RUNNABLE; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:43,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=42, state=RUNNABLE; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:43,241 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=41, state=RUNNABLE; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:43,243 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,244 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583243"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583243"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583243"}]},"ts":"1676289583243"} 2023-02-13 11:59:43,245 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=38, state=RUNNABLE; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:43,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=40, state=RUNNABLE; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:43,250 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=39, state=RUNNABLE; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:43,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 11:59:43,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:43,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 11:59:43,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:43,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:43,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:43,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:43,397 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:43,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:43,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 11:59:43,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:43,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:43,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:43,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:43,399 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:43,399 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:43,400 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:43,400 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81f9e5dd49cc8e0565489f1ebf657e0d columnFamilyName cf 2023-02-13 11:59:43,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:43,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 11:59:43,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:43,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:43,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:43,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:43,402 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(310): Store=81f9e5dd49cc8e0565489f1ebf657e0d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:43,403 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:43,403 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:43,403 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:43,404 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ee51e8905082f3fc07662881dfd80c columnFamilyName cf 2023-02-13 11:59:43,404 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:43,404 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:43,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:43,405 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 56eee98ead755df3cb16e7153d171fbf columnFamilyName cf 2023-02-13 11:59:43,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:43,406 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(310): Store=e0ee51e8905082f3fc07662881dfd80c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:43,406 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(310): Store=56eee98ead755df3cb16e7153d171fbf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:43,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:43,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:43,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:43,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:43,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:43,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81f9e5dd49cc8e0565489f1ebf657e0d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72040393, jitterRate=0.07348550856113434}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:43,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Running coprocessor pre-open hook at 1676289583395Writing region info on filesystem at 1676289583395Initializing all the Stores at 1676289583397 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289583397Cleaning up temporary data from old regions at 1676289583407 (+10 ms)Cleaning up detritus from prior splits at 1676289583407Running coprocessor post-open hooks at 1676289583415 (+8 ms)Region opened successfully at 1676289583415 2023-02-13 11:59:43,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., pid=43, masterSystemTime=1676289583390 2023-02-13 11:59:43,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:43,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:43,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:43,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:43,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:43,421 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened e0ee51e8905082f3fc07662881dfd80c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64261459, jitterRate=-0.042429640889167786}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:43,422 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289583421"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289583421"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289583421"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289583421"}]},"ts":"1676289583421"} 2023-02-13 11:59:43,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 56eee98ead755df3cb16e7153d171fbf; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64032618, jitterRate=-0.04583963751792908}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:43,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 56eee98ead755df3cb16e7153d171fbf: Running coprocessor pre-open hook at 1676289583401Writing region info on filesystem at 1676289583401Initializing all the Stores at 1676289583402 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289583402Cleaning up temporary data from old regions at 1676289583414 (+12 ms)Cleaning up detritus from prior splits at 1676289583415 (+1 ms)Running coprocessor post-open hooks at 1676289583422 (+7 ms)Region opened successfully at 1676289583422 2023-02-13 11:59:43,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 11:59:43,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for e0ee51e8905082f3fc07662881dfd80c: Running coprocessor pre-open hook at 1676289583398Writing region info on filesystem at 1676289583398Initializing all the Stores at 1676289583399 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289583399Cleaning up temporary data from old regions at 1676289583414 (+15 ms)Cleaning up detritus from prior splits at 1676289583415 (+1 ms)Running coprocessor post-open hooks at 1676289583422 (+7 ms)Region opened successfully at 1676289583422 2023-02-13 11:59:43,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:43,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:43,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:43,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:43,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., pid=48, masterSystemTime=1676289583393 2023-02-13 11:59:43,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., pid=46, masterSystemTime=1676289583396 2023-02-13 11:59:43,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:43,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:43,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:43,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 11:59:43,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:43,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:43,428 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,428 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583428"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289583428"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289583428"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289583428"}]},"ts":"1676289583428"} 2023-02-13 11:59:43,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:43,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:43,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:43,429 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:43,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:43,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 11:59:43,430 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:43,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:43,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:43,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:43,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:43,432 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:43,432 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:43,433 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a40bfc382ab282660f54577d5b7b9575 columnFamilyName cf 2023-02-13 11:59:43,434 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(310): Store=a40bfc382ab282660f54577d5b7b9575/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:43,440 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=37 2023-02-13 11:59:43,440 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,440 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=37, state=SUCCESS; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 188 msec 2023-02-13 11:59:43,440 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583440"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289583440"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289583440"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289583440"}]},"ts":"1676289583440"} 2023-02-13 11:59:43,441 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:43,442 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:43,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:43,445 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:43,445 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:43,445 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN in 370 msec 2023-02-13 11:59:43,445 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b55b9ce69d00ead52d760ba43038e9a columnFamilyName cf 2023-02-13 11:59:43,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:43,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=38 2023-02-13 11:59:43,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=38, state=SUCCESS; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 192 msec 2023-02-13 11:59:43,447 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:43,447 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:43,448 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c62ea170afdbf159d444aa319925e012 columnFamilyName cf 2023-02-13 11:59:43,448 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(310): Store=2b55b9ce69d00ead52d760ba43038e9a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:43,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:43,450 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(310): Store=c62ea170afdbf159d444aa319925e012/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:43,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN in 377 msec 2023-02-13 11:59:43,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=39 2023-02-13 11:59:43,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=39, state=SUCCESS; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 195 msec 2023-02-13 11:59:43,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:43,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:43,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:43,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:43,455 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN in 381 msec 2023-02-13 11:59:43,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a40bfc382ab282660f54577d5b7b9575; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60286506, jitterRate=-0.10166105628013611}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:43,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a40bfc382ab282660f54577d5b7b9575: Running coprocessor pre-open hook at 1676289583423Writing region info on filesystem at 1676289583423Initializing all the Stores at 1676289583429 (+6 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289583429Cleaning up temporary data from old regions at 1676289583447 (+18 ms)Cleaning up detritus from prior splits at 1676289583448 (+1 ms)Running coprocessor post-open hooks at 1676289583457 (+9 ms)Region opened successfully at 1676289583457 2023-02-13 11:59:43,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:43,458 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., pid=47, masterSystemTime=1676289583390 2023-02-13 11:59:43,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 2b55b9ce69d00ead52d760ba43038e9a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70763537, jitterRate=0.054458871483802795}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:43,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 2b55b9ce69d00ead52d760ba43038e9a: Running coprocessor pre-open hook at 1676289583431Writing region info on filesystem at 1676289583431Initializing all the Stores at 1676289583432 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289583432Cleaning up temporary data from old regions at 1676289583453 (+21 ms)Cleaning up detritus from prior splits at 1676289583454 (+1 ms)Running coprocessor post-open hooks at 1676289583460 (+6 ms)Region opened successfully at 1676289583460 2023-02-13 11:59:43,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., pid=44, masterSystemTime=1676289583393 2023-02-13 11:59:43,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:43,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:43,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:43,463 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,463 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583463"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289583463"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289583463"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289583463"}]},"ts":"1676289583463"} 2023-02-13 11:59:43,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened c62ea170afdbf159d444aa319925e012; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65809429, jitterRate=-0.01936309039592743}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:43,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for c62ea170afdbf159d444aa319925e012: Running coprocessor pre-open hook at 1676289583429Writing region info on filesystem at 1676289583429Initializing all the Stores at 1676289583431 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289583432 (+1 ms)Cleaning up temporary data from old regions at 1676289583456 (+24 ms)Cleaning up detritus from prior splits at 1676289583456Running coprocessor post-open hooks at 1676289583464 (+8 ms)Region opened successfully at 1676289583464 2023-02-13 11:59:43,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:43,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., pid=45, masterSystemTime=1676289583396 2023-02-13 11:59:43,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:43,467 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,467 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289583467"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289583467"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289583467"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289583467"}]},"ts":"1676289583467"} 2023-02-13 11:59:43,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:43,469 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:43,470 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,471 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583470"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289583470"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289583470"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289583470"}]},"ts":"1676289583470"} 2023-02-13 11:59:43,473 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=40 2023-02-13 11:59:43,473 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=40, state=SUCCESS; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 219 msec 2023-02-13 11:59:43,475 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=42 2023-02-13 11:59:43,475 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=42, state=SUCCESS; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 232 msec 2023-02-13 11:59:43,476 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN in 403 msec 2023-02-13 11:59:43,477 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=41 2023-02-13 11:59:43,477 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN in 405 msec 2023-02-13 11:59:43,477 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=41, state=SUCCESS; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 233 msec 2023-02-13 11:59:43,480 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=36 2023-02-13 11:59:43,480 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN in 407 msec 2023-02-13 11:59:43,481 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289583481"}]},"ts":"1676289583481"} 2023-02-13 11:59:43,483 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLED in hbase:meta 2023-02-13 11:59:43,529 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676289580536' was successfully enabled. 2023-02-13 11:59:43,532 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676289580536 in 513 msec 2023-02-13 11:59:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 11:59:43,783 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 36 completed 2023-02-13 11:59:43,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:43,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:43,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:43,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:43,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:43,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:43,842 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,842 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:43,843 INFO [Listener at localhost.localdomain/36409] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 11:59:43,867 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 11:59:43,869 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 11:59:43,872 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 11:59:43,874 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 11:59:43,876 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 11:59:43,888 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,888 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:43,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 11:59:43,892 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289583892"}]},"ts":"1676289583892"} 2023-02-13 11:59:43,894 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLING in hbase:meta 2023-02-13 11:59:43,947 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676289580536 to state=DISABLING 2023-02-13 11:59:43,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN}] 2023-02-13 11:59:43,951 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN 2023-02-13 11:59:43,951 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN 2023-02-13 11:59:43,951 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN 2023-02-13 11:59:43,951 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN 2023-02-13 11:59:43,951 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN 2023-02-13 11:59:43,952 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,952 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,952 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,952 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583952"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583952"}]},"ts":"1676289583952"} 2023-02-13 11:59:43,952 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583952"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583952"}]},"ts":"1676289583952"} 2023-02-13 11:59:43,952 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289583952"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583952"}]},"ts":"1676289583952"} 2023-02-13 11:59:43,952 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:43,952 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:43,952 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583952"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583952"}]},"ts":"1676289583952"} 2023-02-13 11:59:43,952 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289583952"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583952"}]},"ts":"1676289583952"} 2023-02-13 11:59:43,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=52, state=RUNNABLE; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:43,955 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=51, state=RUNNABLE; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:43,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=50, state=RUNNABLE; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:43,958 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=53, state=RUNNABLE; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:43,959 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=55, state=RUNNABLE; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:43,959 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN 2023-02-13 11:59:43,961 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:43,961 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289583961"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289583961"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289583961"}]},"ts":"1676289583961"} 2023-02-13 11:59:43,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=54, state=RUNNABLE; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:44,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 11:59:44,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 11:59:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:44,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:44,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 2b55b9ce69d00ead52d760ba43038e9a 1/1 column families, dataSize=22.70 KB heapSize=49.05 KB 2023-02-13 11:59:44,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 11:59:44,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:44,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:44,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 11:59:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:44,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing c62ea170afdbf159d444aa319925e012 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-13 11:59:44,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 11:59:44,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:44,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 11:59:44,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:44,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 81f9e5dd49cc8e0565489f1ebf657e0d 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 11:59:44,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 11:59:44,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 11:59:44,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 11:59:44,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 11:59:44,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/.tmp/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:44,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/.tmp/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:44,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/.tmp/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:44,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/.tmp/cf/d847b93f4cdb425190d84b39613c4e23 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:44,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/.tmp/cf/ff22144d81504ead9754a1bdee26e1cf as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:44,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/.tmp/cf/6f4565b390274a5e9c444537d8b9bca3 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:44,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf, entries=347, sequenceid=8, filesize=64.0 K 2023-02-13 11:59:44,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.70 KB/23245, heapSize ~49.03 KB/50208, currentSize=0 B/0 for 2b55b9ce69d00ead52d760ba43038e9a in 257ms, sequenceid=8, compaction requested=false 2023-02-13 11:59:44,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3, entries=38, sequenceid=8, filesize=11.4 K 2023-02-13 11:59:44,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676289580536' 2023-02-13 11:59:44,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23, entries=36, sequenceid=8, filesize=11.0 K 2023-02-13 11:59:44,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for c62ea170afdbf159d444aa319925e012 in 258ms, sequenceid=8, compaction requested=false 2023-02-13 11:59:44,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676289580536' 2023-02-13 11:59:44,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 81f9e5dd49cc8e0565489f1ebf657e0d in 260ms, sequenceid=8, compaction requested=false 2023-02-13 11:59:44,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 11:59:44,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 11:59:44,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:44,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289584109Running coprocessor pre-close hooks at 1676289584109Disabling compacts and flushes for region at 1676289584109Disabling writes for close at 1676289584111 (+2 ms)Obtaining lock to block concurrent updates at 1676289584111Preparing flush snapshotting stores in c62ea170afdbf159d444aa319925e012 at 1676289584111Finished memstore snapshotting testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676289584120 (+9 ms)Flushing stores of testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. at 1676289584121 (+1 ms)Flushing c62ea170afdbf159d444aa319925e012/cf: creating writer at 1676289584126 (+5 ms)Flushing mob file c62ea170afdbf159d444aa319925e012/cf: appending metadata at 1676289584169 (+43 ms)Flushing mob file c62ea170afdbf159d444aa319925e012/cf: closing flushed file at 1676289584170 (+1 ms)Flushing c62ea170afdbf159d444aa319925e012/cf: appending metadata at 1676289584286 (+116 ms)Flushing c62ea170afdbf159d444aa319925e012/cf: closing flushed file at 1676289584294 (+8 ms)Flushing c62ea170afdbf159d444aa319925e012/cf: reopening flushed file at 1676289584359 (+65 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for c62ea170afdbf159d444aa319925e012 in 258ms, sequenceid=8, compaction requested=false at 1676289584369 (+10 ms)Writing region close event to WAL at 1676289584382 (+13 ms)Running coprocessor post-close hooks at 1676289584390 (+8 ms)Closed at 1676289584390 2023-02-13 11:59:44,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:44,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289584111Running coprocessor pre-close hooks at 1676289584111Disabling compacts and flushes for region at 1676289584111Disabling writes for close at 1676289584112 (+1 ms)Obtaining lock to block concurrent updates at 1676289584112Preparing flush snapshotting stores in 81f9e5dd49cc8e0565489f1ebf657e0d at 1676289584112Finished memstore snapshotting testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676289584120 (+8 ms)Flushing stores of testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. at 1676289584123 (+3 ms)Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: creating writer at 1676289584126 (+3 ms)Flushing mob file 81f9e5dd49cc8e0565489f1ebf657e0d/cf: appending metadata at 1676289584167 (+41 ms)Flushing mob file 81f9e5dd49cc8e0565489f1ebf657e0d/cf: closing flushed file at 1676289584167Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: appending metadata at 1676289584291 (+124 ms)Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: closing flushed file at 1676289584294 (+3 ms)Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: reopening flushed file at 1676289584356 (+62 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 81f9e5dd49cc8e0565489f1ebf657e0d in 260ms, sequenceid=8, compaction requested=false at 1676289584372 (+16 ms)Writing region close event to WAL at 1676289584382 (+10 ms)Running coprocessor post-close hooks at 1676289584390 (+8 ms)Closed at 1676289584390 2023-02-13 11:59:44,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 11:59:44,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:44,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289584107Running coprocessor pre-close hooks at 1676289584107Disabling compacts and flushes for region at 1676289584107Disabling writes for close at 1676289584108 (+1 ms)Obtaining lock to block concurrent updates at 1676289584110 (+2 ms)Preparing flush snapshotting stores in 2b55b9ce69d00ead52d760ba43038e9a at 1676289584110Finished memstore snapshotting testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., syncing WAL and waiting on mvcc, flushsize=dataSize=23245, getHeapSize=50208, getOffHeapSize=0, getCellsCount=347 at 1676289584120 (+10 ms)Flushing stores of testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. at 1676289584123 (+3 ms)Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: creating writer at 1676289584126 (+3 ms)Flushing mob file 2b55b9ce69d00ead52d760ba43038e9a/cf: appending metadata at 1676289584183 (+57 ms)Flushing mob file 2b55b9ce69d00ead52d760ba43038e9a/cf: closing flushed file at 1676289584183Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: appending metadata at 1676289584291 (+108 ms)Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: closing flushed file at 1676289584294 (+3 ms)Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: reopening flushed file at 1676289584357 (+63 ms)Finished flush of dataSize ~22.70 KB/23245, heapSize ~49.03 KB/50208, currentSize=0 B/0 for 2b55b9ce69d00ead52d760ba43038e9a in 257ms, sequenceid=8, compaction requested=false at 1676289584366 (+9 ms)Writing region close event to WAL at 1676289584382 (+16 ms)Running coprocessor post-close hooks at 1676289584392 (+10 ms)Closed at 1676289584392 2023-02-13 11:59:44,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:44,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:44,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 11:59:44,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:44,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:44,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 11:59:44,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:44,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 56eee98ead755df3cb16e7153d171fbf 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-13 11:59:44,394 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSED 2023-02-13 11:59:44,394 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289584394"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289584394"}]},"ts":"1676289584394"} 2023-02-13 11:59:44,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:44,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:44,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 11:59:44,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:44,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:44,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 11:59:44,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:44,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing a40bfc382ab282660f54577d5b7b9575 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-13 11:59:44,396 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSED 2023-02-13 11:59:44,397 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289584395"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289584395"}]},"ts":"1676289584395"} 2023-02-13 11:59:44,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:44,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:44,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 11:59:44,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:44,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:44,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 11:59:44,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:44,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing e0ee51e8905082f3fc07662881dfd80c 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-13 11:59:44,401 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSED 2023-02-13 11:59:44,402 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289584401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289584401"}]},"ts":"1676289584401"} 2023-02-13 11:59:44,406 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=54 2023-02-13 11:59:44,406 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=54, state=SUCCESS; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 436 msec 2023-02-13 11:59:44,407 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=50 2023-02-13 11:59:44,407 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN in 458 msec 2023-02-13 11:59:44,407 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=50, state=SUCCESS; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 447 msec 2023-02-13 11:59:44,409 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=55 2023-02-13 11:59:44,409 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=55, state=SUCCESS; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 447 msec 2023-02-13 11:59:44,410 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN in 459 msec 2023-02-13 11:59:44,411 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN in 461 msec 2023-02-13 11:59:44,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 11:59:44,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 11:59:44,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 11:59:44,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/.tmp/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:44,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/.tmp/cf/1e304a838f584997acee189c66a2bea0 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:44,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/.tmp/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:44,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0, entries=26, sequenceid=8, filesize=9.3 K 2023-02-13 11:59:44,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/.tmp/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:44,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 56eee98ead755df3cb16e7153d171fbf in 130ms, sequenceid=8, compaction requested=false 2023-02-13 11:59:44,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/.tmp/cf/276801e0d81b49fd9821e25bfaecd480 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:44,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 11:59:44,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:44,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289584393Running coprocessor pre-close hooks at 1676289584393Disabling compacts and flushes for region at 1676289584393Disabling writes for close at 1676289584394 (+1 ms)Obtaining lock to block concurrent updates at 1676289584394Preparing flush snapshotting stores in 56eee98ead755df3cb16e7153d171fbf at 1676289584394Finished memstore snapshotting testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676289584394Flushing stores of testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. at 1676289584395 (+1 ms)Flushing 56eee98ead755df3cb16e7153d171fbf/cf: creating writer at 1676289584396 (+1 ms)Flushing mob file 56eee98ead755df3cb16e7153d171fbf/cf: appending metadata at 1676289584410 (+14 ms)Flushing mob file 56eee98ead755df3cb16e7153d171fbf/cf: closing flushed file at 1676289584410Flushing 56eee98ead755df3cb16e7153d171fbf/cf: appending metadata at 1676289584450 (+40 ms)Flushing 56eee98ead755df3cb16e7153d171fbf/cf: closing flushed file at 1676289584450Flushing 56eee98ead755df3cb16e7153d171fbf/cf: reopening flushed file at 1676289584508 (+58 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 56eee98ead755df3cb16e7153d171fbf in 130ms, sequenceid=8, compaction requested=false at 1676289584524 (+16 ms)Writing region close event to WAL at 1676289584526 (+2 ms)Running coprocessor post-close hooks at 1676289584534 (+8 ms)Closed at 1676289584534 2023-02-13 11:59:44,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:44,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/.tmp/cf/599dabece3764fa8a0da81bc49ad9d1c as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:44,539 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSED 2023-02-13 11:59:44,539 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289584539"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289584539"}]},"ts":"1676289584539"} 2023-02-13 11:59:44,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480, entries=23, sequenceid=8, filesize=8.8 K 2023-02-13 11:59:44,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for a40bfc382ab282660f54577d5b7b9575 in 146ms, sequenceid=8, compaction requested=false 2023-02-13 11:59:44,545 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=51 2023-02-13 11:59:44,546 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=51, state=SUCCESS; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 586 msec 2023-02-13 11:59:44,550 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN in 598 msec 2023-02-13 11:59:44,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c, entries=30, sequenceid=8, filesize=10.0 K 2023-02-13 11:59:44,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for e0ee51e8905082f3fc07662881dfd80c in 153ms, sequenceid=8, compaction requested=false 2023-02-13 11:59:44,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 11:59:44,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 11:59:44,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:44,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289584395Running coprocessor pre-close hooks at 1676289584395Disabling compacts and flushes for region at 1676289584395Disabling writes for close at 1676289584395Obtaining lock to block concurrent updates at 1676289584395Preparing flush snapshotting stores in a40bfc382ab282660f54577d5b7b9575 at 1676289584395Finished memstore snapshotting testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., syncing WAL and waiting on mvcc, flushsize=dataSize=1539, getHeapSize=3552, getOffHeapSize=0, getCellsCount=23 at 1676289584395Flushing stores of testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. at 1676289584398 (+3 ms)Flushing a40bfc382ab282660f54577d5b7b9575/cf: creating writer at 1676289584399 (+1 ms)Flushing mob file a40bfc382ab282660f54577d5b7b9575/cf: appending metadata at 1676289584408 (+9 ms)Flushing mob file a40bfc382ab282660f54577d5b7b9575/cf: closing flushed file at 1676289584408Flushing a40bfc382ab282660f54577d5b7b9575/cf: appending metadata at 1676289584481 (+73 ms)Flushing a40bfc382ab282660f54577d5b7b9575/cf: closing flushed file at 1676289584481Flushing a40bfc382ab282660f54577d5b7b9575/cf: reopening flushed file at 1676289584531 (+50 ms)Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for a40bfc382ab282660f54577d5b7b9575 in 146ms, sequenceid=8, compaction requested=false at 1676289584541 (+10 ms)Writing region close event to WAL at 1676289584561 (+20 ms)Running coprocessor post-close hooks at 1676289584569 (+8 ms)Closed at 1676289584569 2023-02-13 11:59:44,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:44,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289584400Running coprocessor pre-close hooks at 1676289584400Disabling compacts and flushes for region at 1676289584400Disabling writes for close at 1676289584401 (+1 ms)Obtaining lock to block concurrent updates at 1676289584401Preparing flush snapshotting stores in e0ee51e8905082f3fc07662881dfd80c at 1676289584401Finished memstore snapshotting testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676289584401Flushing stores of testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. at 1676289584403 (+2 ms)Flushing e0ee51e8905082f3fc07662881dfd80c/cf: creating writer at 1676289584404 (+1 ms)Flushing mob file e0ee51e8905082f3fc07662881dfd80c/cf: appending metadata at 1676289584420 (+16 ms)Flushing mob file e0ee51e8905082f3fc07662881dfd80c/cf: closing flushed file at 1676289584420Flushing e0ee51e8905082f3fc07662881dfd80c/cf: appending metadata at 1676289584482 (+62 ms)Flushing e0ee51e8905082f3fc07662881dfd80c/cf: closing flushed file at 1676289584482Flushing e0ee51e8905082f3fc07662881dfd80c/cf: reopening flushed file at 1676289584538 (+56 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for e0ee51e8905082f3fc07662881dfd80c in 153ms, sequenceid=8, compaction requested=false at 1676289584554 (+16 ms)Writing region close event to WAL at 1676289584562 (+8 ms)Running coprocessor post-close hooks at 1676289584569 (+7 ms)Closed at 1676289584569 2023-02-13 11:59:44,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:44,573 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSED 2023-02-13 11:59:44,574 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:44,574 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289584573"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289584573"}]},"ts":"1676289584573"} 2023-02-13 11:59:44,575 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSED 2023-02-13 11:59:44,575 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289584574"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289584574"}]},"ts":"1676289584574"} 2023-02-13 11:59:44,578 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=53 2023-02-13 11:59:44,578 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=53, state=SUCCESS; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 618 msec 2023-02-13 11:59:44,580 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=52 2023-02-13 11:59:44,580 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN in 630 msec 2023-02-13 11:59:44,580 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=52, state=SUCCESS; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 623 msec 2023-02-13 11:59:44,582 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=49 2023-02-13 11:59:44,582 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN in 632 msec 2023-02-13 11:59:44,583 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289584583"}]},"ts":"1676289584583"} 2023-02-13 11:59:44,585 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLED in hbase:meta 2023-02-13 11:59:44,594 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676289580536 to state=DISABLED 2023-02-13 11:59:44,598 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676289580536 in 708 msec 2023-02-13 11:59:44,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 11:59:44,646 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 49 completed 2023-02-13 11:59:44,647 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:44,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289584647 (current time:1676289584647). 2023-02-13 11:59:44,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 11:59:44,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x772fef49 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:44,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ac552a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:44,666 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52048, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:44,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x772fef49 to 127.0.0.1:63172 2023-02-13 11:59:44,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:44,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 11:59:44,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:44,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:44,671 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:44,672 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE; timeout=600000, timestamp=1676290184672 2023-02-13 11:59:44,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:44,672 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676289580536 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 2023-02-13 11:59:44,673 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 11:59:44,673 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 11:59:44,690 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 11:59:44,694 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:44,694 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:44,694 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:44,694 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:44,694 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:44,694 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:44,696 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:44,696 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:44,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:44,698 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:44,698 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:44,698 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:44,698 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:44,698 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:44,699 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c] hfiles 2023-02-13 11:59:44,699 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:44,699 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480] hfiles 2023-02-13 11:59:44,699 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:44,701 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3] hfiles 2023-02-13 11:59:44,701 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:44,701 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23] hfiles 2023-02-13 11:59:44,701 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:44,702 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf] hfiles 2023-02-13 11:59:44,702 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:44,703 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0] hfiles 2023-02-13 11:59:44,703 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:44,706 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352] hfiles 2023-02-13 11:59:44,706 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 11:59:44,706 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 11:59:44,706 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 11:59:44,707 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 11:59:44,707 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 11:59:44,707 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 11:59:44,749 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } as finished. 2023-02-13 11:59:44,749 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676289580536 2023-02-13 11:59:44,750 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289580536 2023-02-13 11:59:44,796 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289580536 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/snaptb0-1676289580536 2023-02-13 11:59:44,816 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 11:59:44,823 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676289580536 of table testRestoreSnapshot-1676289580536 completed 2023-02-13 11:59:44,823 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289580536 2023-02-13 11:59:44,823 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289580536 2023-02-13 11:59:44,824 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676289580536 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 at 1676289584673Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } at 1676289584691 (+18 ms)Consolidate snapshot: snaptb0-1676289580536 at 1676289584749 (+58 ms)Loading Region manifests for snaptb0-1676289580536 at 1676289584749Writing data manifest for snaptb0-1676289580536 at 1676289584761 (+12 ms)Verifying snapshot: snaptb0-1676289580536 at 1676289584781 (+20 ms)Snapshot snaptb0-1676289580536 of table testRestoreSnapshot-1676289580536 completed at 1676289584823 (+42 ms) 2023-02-13 11:59:44,825 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:44,827 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE in 155 msec 2023-02-13 11:59:44,888 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 11:59:44,888 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 11:59:44,923 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 11:59:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 11:59:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 11:59:44,927 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:44,927 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676289580536 2023-02-13 11:59:44,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:44,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 11:59:44,934 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676289580536 2023-02-13 11:59:44,934 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289584934"}]},"ts":"1676289584934"} 2023-02-13 11:59:44,937 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLING in hbase:meta 2023-02-13 11:59:44,999 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 11:59:45,000 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN}] 2023-02-13 11:59:45,005 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN 2023-02-13 11:59:45,005 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN 2023-02-13 11:59:45,005 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN 2023-02-13 11:59:45,005 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN 2023-02-13 11:59:45,006 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:45,006 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:45,006 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN 2023-02-13 11:59:45,006 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 11:59:45,006 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 11:59:45,007 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:45,007 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:45,007 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 11:59:45,007 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 11:59:45,007 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:45,008 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN 2023-02-13 11:59:45,008 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 11:59:45,008 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:45,009 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 11:59:45,157 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 11:59:45,157 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:45,157 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:45,157 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585157"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289585157"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289585157"}]},"ts":"1676289585157"} 2023-02-13 11:59:45,157 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:45,157 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:45,157 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289585157"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289585157"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289585157"}]},"ts":"1676289585157"} 2023-02-13 11:59:45,157 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:45,157 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585157"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289585157"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289585157"}]},"ts":"1676289585157"} 2023-02-13 11:59:45,157 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289585157"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289585157"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289585157"}]},"ts":"1676289585157"} 2023-02-13 11:59:45,158 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585157"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289585157"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289585157"}]},"ts":"1676289585157"} 2023-02-13 11:59:45,159 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=68, state=RUNNABLE; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:45,160 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=69, state=RUNNABLE; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:45,161 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=67, state=RUNNABLE; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:45,162 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=64, state=RUNNABLE; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:45,163 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:45,163 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585163"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289585163"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289585163"}]},"ts":"1676289585163"} 2023-02-13 11:59:45,164 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=65, state=RUNNABLE; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:45,168 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=66, state=RUNNABLE; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:45,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 11:59:45,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:45,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 11:59:45,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:45,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:45,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 11:59:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:45,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:45,318 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:45,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:45,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 11:59:45,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:45,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:45,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:45,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:45,319 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:45,320 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:45,324 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:45,324 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:45,324 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:45,324 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:45,324 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:45,324 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:45,325 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c62ea170afdbf159d444aa319925e012 columnFamilyName cf 2023-02-13 11:59:45,325 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a40bfc382ab282660f54577d5b7b9575 columnFamilyName cf 2023-02-13 11:59:45,325 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b55b9ce69d00ead52d760ba43038e9a columnFamilyName cf 2023-02-13 11:59:45,336 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:45,337 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(310): Store=c62ea170afdbf159d444aa319925e012/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:45,337 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:45,337 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(310): Store=2b55b9ce69d00ead52d760ba43038e9a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:45,338 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:45,338 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(310): Store=a40bfc382ab282660f54577d5b7b9575/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:45,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:45,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:45,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:45,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:45,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:45,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:45,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:45,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:45,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:45,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 2b55b9ce69d00ead52d760ba43038e9a; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62932042, jitterRate=-0.062239497900009155}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:45,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 2b55b9ce69d00ead52d760ba43038e9a: Running coprocessor pre-open hook at 1676289585318Writing region info on filesystem at 1676289585318Initializing all the Stores at 1676289585319 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289585319Cleaning up temporary data from old regions at 1676289585341 (+22 ms)Cleaning up detritus from prior splits at 1676289585342 (+1 ms)Running coprocessor post-open hooks at 1676289585347 (+5 ms)Region opened successfully at 1676289585347 2023-02-13 11:59:45,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a40bfc382ab282660f54577d5b7b9575; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66555298, jitterRate=-0.008248776197433472}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:45,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a40bfc382ab282660f54577d5b7b9575: Running coprocessor pre-open hook at 1676289585319Writing region info on filesystem at 1676289585319Initializing all the Stores at 1676289585320 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289585320Cleaning up temporary data from old regions at 1676289585342 (+22 ms)Cleaning up detritus from prior splits at 1676289585342Running coprocessor post-open hooks at 1676289585347 (+5 ms)Region opened successfully at 1676289585347 2023-02-13 11:59:45,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened c62ea170afdbf159d444aa319925e012; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74891153, jitterRate=0.11596514284610748}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:45,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for c62ea170afdbf159d444aa319925e012: Running coprocessor pre-open hook at 1676289585317Writing region info on filesystem at 1676289585317Initializing all the Stores at 1676289585317Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289585318 (+1 ms)Cleaning up temporary data from old regions at 1676289585342 (+24 ms)Cleaning up detritus from prior splits at 1676289585343 (+1 ms)Running coprocessor post-open hooks at 1676289585348 (+5 ms)Region opened successfully at 1676289585348 2023-02-13 11:59:45,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., pid=71, masterSystemTime=1676289585312 2023-02-13 11:59:45,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., pid=72, masterSystemTime=1676289585315 2023-02-13 11:59:45,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., pid=70, masterSystemTime=1676289585311 2023-02-13 11:59:45,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:45,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:45,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:45,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 11:59:45,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:45,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:45,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:45,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:45,352 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:45,352 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585352"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289585352"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289585352"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289585352"}]},"ts":"1676289585352"} 2023-02-13 11:59:45,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:45,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:45,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:45,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 11:59:45,354 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:45,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:45,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:45,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:45,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 11:59:45,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:45,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:45,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:45,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:45,354 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289585354"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289585354"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289585354"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289585354"}]},"ts":"1676289585354"} 2023-02-13 11:59:45,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:45,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:45,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:45,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:45,356 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:45,357 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585356"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289585356"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289585356"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289585356"}]},"ts":"1676289585356"} 2023-02-13 11:59:45,355 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:45,358 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:45,359 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:45,359 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:45,359 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:45,360 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:45,360 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:45,360 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81f9e5dd49cc8e0565489f1ebf657e0d columnFamilyName cf 2023-02-13 11:59:45,360 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 56eee98ead755df3cb16e7153d171fbf columnFamilyName cf 2023-02-13 11:59:45,361 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:45,361 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:45,361 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ee51e8905082f3fc07662881dfd80c columnFamilyName cf 2023-02-13 11:59:45,365 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=67 2023-02-13 11:59:45,365 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=67, state=SUCCESS; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 197 msec 2023-02-13 11:59:45,366 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=69 2023-02-13 11:59:45,366 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=69, state=SUCCESS; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 202 msec 2023-02-13 11:59:45,368 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN in 365 msec 2023-02-13 11:59:45,368 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=68 2023-02-13 11:59:45,368 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=68, state=SUCCESS; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 204 msec 2023-02-13 11:59:45,369 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN in 366 msec 2023-02-13 11:59:45,373 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:45,373 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(310): Store=81f9e5dd49cc8e0565489f1ebf657e0d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:45,373 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:45,373 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(310): Store=56eee98ead755df3cb16e7153d171fbf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:45,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:45,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:45,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:45,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN in 368 msec 2023-02-13 11:59:45,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:45,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:45,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:45,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 56eee98ead755df3cb16e7153d171fbf; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64400791, jitterRate=-0.040353432297706604}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:45,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81f9e5dd49cc8e0565489f1ebf657e0d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69182108, jitterRate=0.03089374303817749}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:45,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 56eee98ead755df3cb16e7153d171fbf: Running coprocessor pre-open hook at 1676289585355Writing region info on filesystem at 1676289585355Initializing all the Stores at 1676289585357 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289585357Cleaning up temporary data from old regions at 1676289585376 (+19 ms)Cleaning up detritus from prior splits at 1676289585377 (+1 ms)Running coprocessor post-open hooks at 1676289585381 (+4 ms)Region opened successfully at 1676289585381 2023-02-13 11:59:45,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Running coprocessor pre-open hook at 1676289585352Writing region info on filesystem at 1676289585352Initializing all the Stores at 1676289585354 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289585355 (+1 ms)Cleaning up temporary data from old regions at 1676289585377 (+22 ms)Cleaning up detritus from prior splits at 1676289585377Running coprocessor post-open hooks at 1676289585381 (+4 ms)Region opened successfully at 1676289585381 2023-02-13 11:59:45,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., pid=73, masterSystemTime=1676289585315 2023-02-13 11:59:45,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., pid=74, masterSystemTime=1676289585311 2023-02-13 11:59:45,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:45,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:45,384 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:45,384 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585384"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289585384"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289585384"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289585384"}]},"ts":"1676289585384"} 2023-02-13 11:59:45,385 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:45,385 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(310): Store=e0ee51e8905082f3fc07662881dfd80c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:45,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:45,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:45,386 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:45,386 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289585386"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289585386"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289585386"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289585386"}]},"ts":"1676289585386"} 2023-02-13 11:59:45,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:45,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:45,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=65 2023-02-13 11:59:45,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=65, state=SUCCESS; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 222 msec 2023-02-13 11:59:45,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=64 2023-02-13 11:59:45,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=64, state=SUCCESS; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 226 msec 2023-02-13 11:59:45,391 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN in 390 msec 2023-02-13 11:59:45,392 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN in 391 msec 2023-02-13 11:59:45,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:45,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened e0ee51e8905082f3fc07662881dfd80c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58827866, jitterRate=-0.12339648604393005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:45,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for e0ee51e8905082f3fc07662881dfd80c: Running coprocessor pre-open hook at 1676289585356Writing region info on filesystem at 1676289585356Initializing all the Stores at 1676289585358 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289585358Cleaning up temporary data from old regions at 1676289585389 (+31 ms)Cleaning up detritus from prior splits at 1676289585390 (+1 ms)Running coprocessor post-open hooks at 1676289585394 (+4 ms)Region opened successfully at 1676289585394 2023-02-13 11:59:45,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., pid=75, masterSystemTime=1676289585312 2023-02-13 11:59:45,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:45,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:45,397 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:45,398 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289585397"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289585397"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289585397"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289585397"}]},"ts":"1676289585397"} 2023-02-13 11:59:45,401 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=66 2023-02-13 11:59:45,402 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=66, state=SUCCESS; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 233 msec 2023-02-13 11:59:45,403 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=66, resume processing ppid=63 2023-02-13 11:59:45,404 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN in 401 msec 2023-02-13 11:59:45,404 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289585404"}]},"ts":"1676289585404"} 2023-02-13 11:59:45,406 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLED in hbase:meta 2023-02-13 11:59:45,416 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676289580536' was successfully enabled. 2023-02-13 11:59:45,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676289580536 in 490 msec 2023-02-13 11:59:45,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 11:59:45,686 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 63 completed 2023-02-13 11:59:45,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:45,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:45,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:45,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:45,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:45,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 11:59:45,714 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676289580536 2023-02-13 11:59:45,714 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:45,716 INFO [Listener at localhost.localdomain/36409] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 11:59:46,035 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 11:59:46,182 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 11:59:46,346 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 11:59:46,468 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 11:59:46,479 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 11:59:46,480 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 11:59:46,684 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 11:59:48,486 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(139): Connect 0x38e37a0c to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:48,512 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@520116e3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:48,516 DEBUG [hconnection-0x3bd0b098-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:48,526 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52056, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:48,684 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 11:59:48,687 DEBUG [hconnection-0x3bd0b098-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:48,690 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57658, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:48,826 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 11:59:48,834 DEBUG [hconnection-0x3bd0b098-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:48,839 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48700, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:48,990 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 11:59:49,110 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 11:59:49,266 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 11:59:49,893 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 11:59:50,747 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:50,748 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676289580536 2023-02-13 11:59:50,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:50,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 11:59:50,755 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289590755"}]},"ts":"1676289590755"} 2023-02-13 11:59:50,757 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLING in hbase:meta 2023-02-13 11:59:50,792 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676289580536 to state=DISABLING 2023-02-13 11:59:50,793 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN}] 2023-02-13 11:59:50,795 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN 2023-02-13 11:59:50,796 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN 2023-02-13 11:59:50,796 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN 2023-02-13 11:59:50,797 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN 2023-02-13 11:59:50,797 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN 2023-02-13 11:59:50,797 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:50,798 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:50,798 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289590797"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289590797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289590797"}]},"ts":"1676289590797"} 2023-02-13 11:59:50,798 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:50,798 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289590798"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289590798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289590798"}]},"ts":"1676289590798"} 2023-02-13 11:59:50,798 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289590798"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289590798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289590798"}]},"ts":"1676289590798"} 2023-02-13 11:59:50,798 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:50,798 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:50,798 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289590798"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289590798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289590798"}]},"ts":"1676289590798"} 2023-02-13 11:59:50,798 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289590798"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289590798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289590798"}]},"ts":"1676289590798"} 2023-02-13 11:59:50,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=82, state=RUNNABLE; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:50,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=80, state=RUNNABLE; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:50,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=78, state=RUNNABLE; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:50,804 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=79, state=RUNNABLE; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:50,805 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN 2023-02-13 11:59:50,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=81, state=RUNNABLE; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:50,808 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:50,808 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289590808"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289590808"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289590808"}]},"ts":"1676289590808"} 2023-02-13 11:59:50,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=77, state=RUNNABLE; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:50,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:50,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 11:59:50,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:50,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:50,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 11:59:50,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:50,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 2b55b9ce69d00ead52d760ba43038e9a 1/1 column families, dataSize=21.98 KB heapSize=47.50 KB 2023-02-13 11:59:50,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:50,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 11:59:50,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:50,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:50,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 11:59:50,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:50,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 81f9e5dd49cc8e0565489f1ebf657e0d 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-13 11:59:50,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 11:59:50,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 11:59:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:50,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 56eee98ead755df3cb16e7153d171fbf 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-13 11:59:51,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 11:59:51,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 2023-02-13 11:59:51,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea 2023-02-13 11:59:51,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d 2023-02-13 11:59:51,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/.tmp/cf/29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 11:59:51,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/.tmp/cf/29f3eaf3b2c946079123a6bd68c2201a as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 11:59:51,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a, entries=336, sequenceid=15, filesize=62.2 K 2023-02-13 11:59:51,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 2b55b9ce69d00ead52d760ba43038e9a in 87ms, sequenceid=15, compaction requested=false 2023-02-13 11:59:51,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/.tmp/cf/efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 11:59:51,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/.tmp/cf/efd9c8e078ce4850b8dcbe3b73c8a105 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 11:59:51,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105, entries=27, sequenceid=15, filesize=9.5 K 2023-02-13 11:59:51,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 56eee98ead755df3cb16e7153d171fbf in 103ms, sequenceid=15, compaction requested=false 2023-02-13 11:59:51,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/.tmp/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 11:59:51,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 11:59:51,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 11:59:51,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:51,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289590952Running coprocessor pre-close hooks at 1676289590952Disabling compacts and flushes for region at 1676289590952Disabling writes for close at 1676289590953 (+1 ms)Obtaining lock to block concurrent updates at 1676289590954 (+1 ms)Preparing flush snapshotting stores in 2b55b9ce69d00ead52d760ba43038e9a at 1676289590954Finished memstore snapshotting testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., syncing WAL and waiting on mvcc, flushsize=dataSize=22508, getHeapSize=48624, getOffHeapSize=0, getCellsCount=336 at 1676289590954Flushing stores of testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. at 1676289590954Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: creating writer at 1676289590955 (+1 ms)Flushing mob file 2b55b9ce69d00ead52d760ba43038e9a/cf: appending metadata at 1676289590970 (+15 ms)Flushing mob file 2b55b9ce69d00ead52d760ba43038e9a/cf: closing flushed file at 1676289590970Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: appending metadata at 1676289591009 (+39 ms)Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: closing flushed file at 1676289591009Flushing 2b55b9ce69d00ead52d760ba43038e9a/cf: reopening flushed file at 1676289591035 (+26 ms)Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for 2b55b9ce69d00ead52d760ba43038e9a in 87ms, sequenceid=15, compaction requested=false at 1676289591040 (+5 ms)Writing region close event to WAL at 1676289591063 (+23 ms)Running coprocessor post-close hooks at 1676289591070 (+7 ms)Closed at 1676289591071 (+1 ms) 2023-02-13 11:59:51,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:51,077 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289590959Running coprocessor pre-close hooks at 1676289590959Disabling compacts and flushes for region at 1676289590959Disabling writes for close at 1676289590960 (+1 ms)Obtaining lock to block concurrent updates at 1676289590960Preparing flush snapshotting stores in 56eee98ead755df3cb16e7153d171fbf at 1676289590960Finished memstore snapshotting testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676289590960Flushing stores of testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. at 1676289590961 (+1 ms)Flushing 56eee98ead755df3cb16e7153d171fbf/cf: creating writer at 1676289590962 (+1 ms)Flushing mob file 56eee98ead755df3cb16e7153d171fbf/cf: appending metadata at 1676289590977 (+15 ms)Flushing mob file 56eee98ead755df3cb16e7153d171fbf/cf: closing flushed file at 1676289590977Flushing 56eee98ead755df3cb16e7153d171fbf/cf: appending metadata at 1676289591018 (+41 ms)Flushing 56eee98ead755df3cb16e7153d171fbf/cf: closing flushed file at 1676289591018Flushing 56eee98ead755df3cb16e7153d171fbf/cf: reopening flushed file at 1676289591057 (+39 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 56eee98ead755df3cb16e7153d171fbf in 103ms, sequenceid=15, compaction requested=false at 1676289591064 (+7 ms)Writing region close event to WAL at 1676289591067 (+3 ms)Running coprocessor post-close hooks at 1676289591076 (+9 ms)Closed at 1676289591077 (+1 ms) 2023-02-13 11:59:51,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/.tmp/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 11:59:51,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:51,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:51,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 11:59:51,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:51,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:51,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 11:59:51,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:51,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing e0ee51e8905082f3fc07662881dfd80c 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-13 11:59:51,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca, entries=29, sequenceid=15, filesize=9.8 K 2023-02-13 11:59:51,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 81f9e5dd49cc8e0565489f1ebf657e0d in 129ms, sequenceid=15, compaction requested=false 2023-02-13 11:59:51,089 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSED 2023-02-13 11:59:51,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:51,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:51,092 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289591089"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289591089"}]},"ts":"1676289591089"} 2023-02-13 11:59:51,094 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSED 2023-02-13 11:59:51,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 11:59:51,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:51,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:51,094 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289591093"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289591093"}]},"ts":"1676289591093"} 2023-02-13 11:59:51,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 11:59:51,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:51,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing c62ea170afdbf159d444aa319925e012 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-13 11:59:51,101 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=78 2023-02-13 11:59:51,101 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=78, state=SUCCESS; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 296 msec 2023-02-13 11:59:51,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=82 2023-02-13 11:59:51,102 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN in 308 msec 2023-02-13 11:59:51,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=82, state=SUCCESS; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 299 msec 2023-02-13 11:59:51,103 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN in 309 msec 2023-02-13 11:59:51,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 11:59:51,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:51,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289590956Running coprocessor pre-close hooks at 1676289590956Disabling compacts and flushes for region at 1676289590956Disabling writes for close at 1676289590956Obtaining lock to block concurrent updates at 1676289590956Preparing flush snapshotting stores in 81f9e5dd49cc8e0565489f1ebf657e0d at 1676289590956Finished memstore snapshotting testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676289590957 (+1 ms)Flushing stores of testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. at 1676289590957Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: creating writer at 1676289590958 (+1 ms)Flushing mob file 81f9e5dd49cc8e0565489f1ebf657e0d/cf: appending metadata at 1676289590977 (+19 ms)Flushing mob file 81f9e5dd49cc8e0565489f1ebf657e0d/cf: closing flushed file at 1676289590977Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: appending metadata at 1676289591020 (+43 ms)Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: closing flushed file at 1676289591020Flushing 81f9e5dd49cc8e0565489f1ebf657e0d/cf: reopening flushed file at 1676289591080 (+60 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 81f9e5dd49cc8e0565489f1ebf657e0d in 129ms, sequenceid=15, compaction requested=false at 1676289591085 (+5 ms)Writing region close event to WAL at 1676289591094 (+9 ms)Running coprocessor post-close hooks at 1676289591113 (+19 ms)Closed at 1676289591115 (+2 ms) 2023-02-13 11:59:51,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:51,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:51,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 11:59:51,118 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSED 2023-02-13 11:59:51,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:51,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:51,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 11:59:51,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:51,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing a40bfc382ab282660f54577d5b7b9575 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 11:59:51,118 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289591118"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289591118"}]},"ts":"1676289591118"} 2023-02-13 11:59:51,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=77 2023-02-13 11:59:51,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=77, state=SUCCESS; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 310 msec 2023-02-13 11:59:51,123 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN in 328 msec 2023-02-13 11:59:51,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 2023-02-13 11:59:51,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/.tmp/cf/a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 11:59:51,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef 2023-02-13 11:59:51,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/.tmp/cf/a5cf918d60df457f8c1a52e0029a35e4 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 11:59:51,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4, entries=37, sequenceid=15, filesize=11.2 K 2023-02-13 11:59:51,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for c62ea170afdbf159d444aa319925e012 in 80ms, sequenceid=15, compaction requested=false 2023-02-13 11:59:51,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/.tmp/cf/d71da190e77346458dd9bdfc6068675b 2023-02-13 11:59:51,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 11:59:51,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:51,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289591093Running coprocessor pre-close hooks at 1676289591093Disabling compacts and flushes for region at 1676289591093Disabling writes for close at 1676289591095 (+2 ms)Obtaining lock to block concurrent updates at 1676289591095Preparing flush snapshotting stores in c62ea170afdbf159d444aa319925e012 at 1676289591095Finished memstore snapshotting testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676289591095Flushing stores of testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. at 1676289591096 (+1 ms)Flushing c62ea170afdbf159d444aa319925e012/cf: creating writer at 1676289591097 (+1 ms)Flushing mob file c62ea170afdbf159d444aa319925e012/cf: appending metadata at 1676289591117 (+20 ms)Flushing mob file c62ea170afdbf159d444aa319925e012/cf: closing flushed file at 1676289591117Flushing c62ea170afdbf159d444aa319925e012/cf: appending metadata at 1676289591149 (+32 ms)Flushing c62ea170afdbf159d444aa319925e012/cf: closing flushed file at 1676289591149Flushing c62ea170afdbf159d444aa319925e012/cf: reopening flushed file at 1676289591167 (+18 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for c62ea170afdbf159d444aa319925e012 in 80ms, sequenceid=15, compaction requested=false at 1676289591175 (+8 ms)Writing region close event to WAL at 1676289591179 (+4 ms)Running coprocessor post-close hooks at 1676289591184 (+5 ms)Closed at 1676289591185 (+1 ms) 2023-02-13 11:59:51,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:51,187 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSED 2023-02-13 11:59:51,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/.tmp/cf/d71da190e77346458dd9bdfc6068675b as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b 2023-02-13 11:59:51,187 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289591187"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289591187"}]},"ts":"1676289591187"} 2023-02-13 11:59:51,191 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=81 2023-02-13 11:59:51,191 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=81, state=SUCCESS; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 384 msec 2023-02-13 11:59:51,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b, entries=32, sequenceid=15, filesize=10.4 K 2023-02-13 11:59:51,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for a40bfc382ab282660f54577d5b7b9575 in 76ms, sequenceid=15, compaction requested=false 2023-02-13 11:59:51,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN in 398 msec 2023-02-13 11:59:51,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 11:59:51,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:51,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289591118Running coprocessor pre-close hooks at 1676289591118Disabling compacts and flushes for region at 1676289591118Disabling writes for close at 1676289591118Obtaining lock to block concurrent updates at 1676289591118Preparing flush snapshotting stores in a40bfc382ab282660f54577d5b7b9575 at 1676289591118Finished memstore snapshotting testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676289591119 (+1 ms)Flushing stores of testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. at 1676289591120 (+1 ms)Flushing a40bfc382ab282660f54577d5b7b9575/cf: creating writer at 1676289591129 (+9 ms)Flushing mob file a40bfc382ab282660f54577d5b7b9575/cf: appending metadata at 1676289591140 (+11 ms)Flushing mob file a40bfc382ab282660f54577d5b7b9575/cf: closing flushed file at 1676289591140Flushing a40bfc382ab282660f54577d5b7b9575/cf: appending metadata at 1676289591164 (+24 ms)Flushing a40bfc382ab282660f54577d5b7b9575/cf: closing flushed file at 1676289591164Flushing a40bfc382ab282660f54577d5b7b9575/cf: reopening flushed file at 1676289591188 (+24 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for a40bfc382ab282660f54577d5b7b9575 in 76ms, sequenceid=15, compaction requested=false at 1676289591194 (+6 ms)Writing region close event to WAL at 1676289591199 (+5 ms)Running coprocessor post-close hooks at 1676289591204 (+5 ms)Closed at 1676289591204 2023-02-13 11:59:51,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:51,207 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSED 2023-02-13 11:59:51,207 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289591206"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289591206"}]},"ts":"1676289591206"} 2023-02-13 11:59:51,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=80 2023-02-13 11:59:51,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=80, state=SUCCESS; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 407 msec 2023-02-13 11:59:51,212 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN in 417 msec 2023-02-13 11:59:51,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 11:59:51,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 2023-02-13 11:59:51,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/.tmp/cf/c193a7455d3d48f29f60b9563d1f1627 2023-02-13 11:59:51,579 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/.tmp/cf/c193a7455d3d48f29f60b9563d1f1627 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627 2023-02-13 11:59:51,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627, entries=39, sequenceid=15, filesize=11.5 K 2023-02-13 11:59:51,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for e0ee51e8905082f3fc07662881dfd80c in 505ms, sequenceid=15, compaction requested=false 2023-02-13 11:59:51,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 11:59:51,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:51,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289591078Running coprocessor pre-close hooks at 1676289591078Disabling compacts and flushes for region at 1676289591078Disabling writes for close at 1676289591079 (+1 ms)Obtaining lock to block concurrent updates at 1676289591079Preparing flush snapshotting stores in e0ee51e8905082f3fc07662881dfd80c at 1676289591079Finished memstore snapshotting testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676289591080 (+1 ms)Flushing stores of testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. at 1676289591090 (+10 ms)Flushing e0ee51e8905082f3fc07662881dfd80c/cf: creating writer at 1676289591091 (+1 ms)Flushing mob file e0ee51e8905082f3fc07662881dfd80c/cf: appending metadata at 1676289591112 (+21 ms)Flushing mob file e0ee51e8905082f3fc07662881dfd80c/cf: closing flushed file at 1676289591112Flushing e0ee51e8905082f3fc07662881dfd80c/cf: appending metadata at 1676289591556 (+444 ms)Flushing e0ee51e8905082f3fc07662881dfd80c/cf: closing flushed file at 1676289591556Flushing e0ee51e8905082f3fc07662881dfd80c/cf: reopening flushed file at 1676289591580 (+24 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for e0ee51e8905082f3fc07662881dfd80c in 505ms, sequenceid=15, compaction requested=false at 1676289591584 (+4 ms)Writing region close event to WAL at 1676289591593 (+9 ms)Running coprocessor post-close hooks at 1676289591597 (+4 ms)Closed at 1676289591598 (+1 ms) 2023-02-13 11:59:51,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:51,600 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSED 2023-02-13 11:59:51,600 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289591600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289591600"}]},"ts":"1676289591600"} 2023-02-13 11:59:51,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=79 2023-02-13 11:59:51,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=79, state=SUCCESS; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 797 msec 2023-02-13 11:59:51,604 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=79, resume processing ppid=76 2023-02-13 11:59:51,604 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN in 810 msec 2023-02-13 11:59:51,605 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289591605"}]},"ts":"1676289591605"} 2023-02-13 11:59:51,606 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLED in hbase:meta 2023-02-13 11:59:51,702 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676289580536 to state=DISABLED 2023-02-13 11:59:51,704 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676289580536 in 955 msec 2023-02-13 11:59:52,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 11:59:52,258 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 76 completed 2023-02-13 11:59:52,259 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:52,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289592259 (current time:1676289592259). 2023-02-13 11:59:52,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 11:59:52,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x2be68c5f to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:52,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23bb7bcd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:52,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:52,304 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42248, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2be68c5f to 127.0.0.1:63172 2023-02-13 11:59:52,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:52,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 11:59:52,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:52,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:52,310 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:52,311 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE; timeout=600000, timestamp=1676290192311 2023-02-13 11:59:52,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:52,311 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676289580536 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 2023-02-13 11:59:52,312 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 11:59:52,312 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 11:59:52,321 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,324 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,323 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,327 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105] hfiles 2023-02-13 11:59:52,327 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:52,328 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 11:59:52,329 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4] hfiles 2023-02-13 11:59:52,329 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:52,329 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 11:59:52,330 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627] hfiles 2023-02-13 11:59:52,330 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:52,330 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627 2023-02-13 11:59:52,330 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf] hfiles 2023-02-13 11:59:52,330 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 11:59:52,330 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:52,332 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b] hfiles 2023-02-13 11:59:52,332 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:52,332 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b 2023-02-13 11:59:52,333 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca] hfiles 2023-02-13 11:59:52,333 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:52,333 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef] hfiles 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 11:59:52,341 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 11:59:52,342 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef 2023-02-13 11:59:52,374 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } as finished. 2023-02-13 11:59:52,374 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676289580536 2023-02-13 11:59:52,375 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb1-1676289580536 2023-02-13 11:59:52,401 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb1-1676289580536 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/snaptb1-1676289580536 2023-02-13 11:59:52,418 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676289580536 of table testRestoreSnapshot-1676289580536 completed 2023-02-13 11:59:52,418 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb1-1676289580536 2023-02-13 11:59:52,419 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb1-1676289580536 2023-02-13 11:59:52,419 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676289580536 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 at 1676289592311Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } at 1676289592321 (+10 ms)Consolidate snapshot: snaptb1-1676289580536 at 1676289592374 (+53 ms)Loading Region manifests for snaptb1-1676289580536 at 1676289592374Writing data manifest for snaptb1-1676289580536 at 1676289592380 (+6 ms)Verifying snapshot: snaptb1-1676289580536 at 1676289592389 (+9 ms)Snapshot snaptb1-1676289580536 of table testRestoreSnapshot-1676289580536 completed at 1676289592418 (+29 ms) 2023-02-13 11:59:52,420 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:52,421 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE in 112 msec 2023-02-13 11:59:52,562 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 11:59:52,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 11:59:52,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 11:59:52,583 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:52,584 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:52,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289592584 (current time:1676289592584). 2023-02-13 11:59:52,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 11:59:52,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x12875a41 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:52,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1550f8ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:52,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:52,607 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42254, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12875a41 to 127.0.0.1:63172 2023-02-13 11:59:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 11:59:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:52,617 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:52,617 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE; timeout=600000, timestamp=1676290192617 2023-02-13 11:59:52,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:52,618 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676289580536-1676289592583 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 2023-02-13 11:59:52,618 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 11:59:52,618 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 11:59:52,628 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 11:59:52,629 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,630 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,630 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,630 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,630 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,630 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,633 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,634 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,634 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,634 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,634 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,634 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:52,636 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,634 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:52,636 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b] hfiles 2023-02-13 11:59:52,636 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:52,636 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b 2023-02-13 11:59:52,636 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627] hfiles 2023-02-13 11:59:52,636 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:52,636 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627 2023-02-13 11:59:52,637 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca] hfiles 2023-02-13 11:59:52,637 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:52,637 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 11:59:52,637 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf] hfiles 2023-02-13 11:59:52,637 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 11:59:52,637 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:52,639 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4] hfiles 2023-02-13 11:59:52,639 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105] hfiles 2023-02-13 11:59:52,639 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:52,639 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:52,639 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 11:59:52,639 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 11:59:52,644 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef] hfiles 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 11:59:52,645 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef 2023-02-13 11:59:52,868 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 11:59:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 11:59:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 11:59:52,870 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 11:59:53,066 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } as finished. 2023-02-13 11:59:53,066 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:53,067 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:53,104 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676289580536-1676289592583 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:53,131 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676289580536-1676289592583 of table testRestoreSnapshot-1676289580536 completed 2023-02-13 11:59:53,131 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:53,131 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:53,131 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676289580536-1676289592583 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 at 1676289592618Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } at 1676289592628 (+10 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676289580536-1676289592583 at 1676289593066 (+438 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676289580536-1676289592583 at 1676289593067 (+1 ms)Writing data manifest for hbase-failsafe-snaptb0-1676289580536-1676289592583 at 1676289593074 (+7 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676289580536-1676289592583 at 1676289593088 (+14 ms)Snapshot hbase-failsafe-snaptb0-1676289580536-1676289592583 of table testRestoreSnapshot-1676289580536 completed at 1676289593131 (+43 ms) 2023-02-13 11:59:53,133 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:53,134 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE in 518 msec 2023-02-13 11:59:53,370 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 11:59:53,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 11:59:53,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676289580536-1676289592583 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 11:59:53,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "snaptb0-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289584647 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 11:59:53,395 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676289580536 as table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:53,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 11:59:53,438 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000002 2023-02-13 11:59:53,444 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000001 2023-02-13 11:59:53,444 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000002 2023-02-13 11:59:53,445 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 11:59:53,452 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289584647 type: DISABLED version: 2 ttl: 0 2023-02-13 11:59:53,453 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536 2023-02-13 11:59:53,462 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:53,471 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 2023-02-13 11:59:53,473 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d 2023-02-13 11:59:53,474 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea 2023-02-13 11:59:53,476 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef 2023-02-13 11:59:53,477 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 2023-02-13 11:59:53,479 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 2023-02-13 11:59:53,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:53,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:53,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:53,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:53,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:53,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:53,497 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 11:59:53,497 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 11:59:53,497 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 11:59:53,497 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 11:59:53,498 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/c193a7455d3d48f29f60b9563d1f1627 2023-02-13 11:59:53,498 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/d71da190e77346458dd9bdfc6068675b 2023-02-13 11:59:53,498 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289584647 type: DISABLED version: 2 ttl: 0 2023-02-13 11:59:53,500 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289593499"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:53,500 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289593499"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:53,500 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289593499"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:53,500 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289593499"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:53,500 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289593499"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:53,500 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289593499"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:53,503 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 11:59:53,503 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-13 11:59:53,503 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289593500"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289593500"}]},"ts":"1676289593500"} 2023-02-13 11:59:53,503 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593500"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289593500"}]},"ts":"1676289593500"} 2023-02-13 11:59:53,503 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289593500"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289593500"}]},"ts":"1676289593500"} 2023-02-13 11:59:53,503 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593500"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289593500"}]},"ts":"1676289593500"} 2023-02-13 11:59:53,503 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593500"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289593500"}]},"ts":"1676289593500"} 2023-02-13 11:59:53,504 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593500"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289593500"}]},"ts":"1676289593500"} 2023-02-13 11:59:53,512 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 11:59:53,512 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 11:59:53,512 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-13 11:59:53,513 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676289580536 completed! 2023-02-13 11:59:53,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "snaptb0-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289584647 type: DISABLED version: 2 ttl: 0 ) in 127 msec 2023-02-13 11:59:53,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 11:59:53,649 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676289580536, procId: 91 completed 2023-02-13 11:59:53,649 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:53,654 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "hbase-failsafe-snaptb0-1676289580536-1676289592583" 2023-02-13 11:59:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676289580536-1676289592583 2023-02-13 11:59:53,659 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:53,659 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676289580536 2023-02-13 11:59:53,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 11:59:53,663 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676289580536 2023-02-13 11:59:53,663 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289593663"}]},"ts":"1676289593663"} 2023-02-13 11:59:53,664 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLING in hbase:meta 2023-02-13 11:59:53,675 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 11:59:53,677 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN}] 2023-02-13 11:59:53,679 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN 2023-02-13 11:59:53,680 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN 2023-02-13 11:59:53,680 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN 2023-02-13 11:59:53,680 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN 2023-02-13 11:59:53,681 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN 2023-02-13 11:59:53,681 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:53,681 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:53,681 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:53,681 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:53,682 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:53,683 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN 2023-02-13 11:59:53,684 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:53,831 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 11:59:53,832 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:59:53,832 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 11:59:53,832 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 11:59:53,832 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 11:59:53,838 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:53,838 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:53,838 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:53,838 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593838"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289593838"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289593838"}]},"ts":"1676289593838"} 2023-02-13 11:59:53,838 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289593838"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289593838"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289593838"}]},"ts":"1676289593838"} 2023-02-13 11:59:53,838 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:53,838 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:53,839 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289593838"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289593838"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289593838"}]},"ts":"1676289593838"} 2023-02-13 11:59:53,839 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593838"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289593838"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289593838"}]},"ts":"1676289593838"} 2023-02-13 11:59:53,845 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=96, state=RUNNABLE; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:53,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=98, state=RUNNABLE; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:53,838 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593838"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289593838"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289593838"}]},"ts":"1676289593838"} 2023-02-13 11:59:53,849 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=93, state=RUNNABLE; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:53,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=97, state=RUNNABLE; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:53,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=95, state=RUNNABLE; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:53,859 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:53,859 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289593859"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289593859"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289593859"}]},"ts":"1676289593859"} 2023-02-13 11:59:53,862 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=94, state=RUNNABLE; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:53,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 11:59:54,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:54,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 11:59:54,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:54,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:54,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:54,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:54,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:54,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 11:59:54,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:54,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:54,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:54,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:54,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:54,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 11:59:54,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:54,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:54,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:54,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:54,024 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:54,026 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:54,027 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:54,028 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:54,028 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:54,028 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:54,028 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:54,028 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:54,028 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:54,028 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c62ea170afdbf159d444aa319925e012 columnFamilyName cf 2023-02-13 11:59:54,028 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ee51e8905082f3fc07662881dfd80c columnFamilyName cf 2023-02-13 11:59:54,031 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81f9e5dd49cc8e0565489f1ebf657e0d columnFamilyName cf 2023-02-13 11:59:54,042 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:54,044 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(310): Store=c62ea170afdbf159d444aa319925e012/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:54,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:54,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:54,049 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:54,049 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(310): Store=81f9e5dd49cc8e0565489f1ebf657e0d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:54,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:54,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:54,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:54,057 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened c62ea170afdbf159d444aa319925e012; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69633144, jitterRate=0.03761470317840576}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:54,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for c62ea170afdbf159d444aa319925e012: Running coprocessor pre-open hook at 1676289594014Writing region info on filesystem at 1676289594014Initializing all the Stores at 1676289594015 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289594015Cleaning up temporary data from old regions at 1676289594048 (+33 ms)Cleaning up detritus from prior splits at 1676289594048Running coprocessor post-open hooks at 1676289594057 (+9 ms)Region opened successfully at 1676289594057 2023-02-13 11:59:54,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., pid=102, masterSystemTime=1676289594008 2023-02-13 11:59:54,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:54,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:54,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:54,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 11:59:54,061 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:54,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:54,061 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(310): Store=e0ee51e8905082f3fc07662881dfd80c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:54,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:54,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:54,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:54,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:54,062 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:54,062 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289594062"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289594062"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289594062"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289594062"}]},"ts":"1676289594062"} 2023-02-13 11:59:54,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:54,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81f9e5dd49cc8e0565489f1ebf657e0d; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65305571, jitterRate=-0.02687115967273712}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:54,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Running coprocessor pre-open hook at 1676289594010Writing region info on filesystem at 1676289594010Initializing all the Stores at 1676289594011 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289594012 (+1 ms)Cleaning up temporary data from old regions at 1676289594058 (+46 ms)Cleaning up detritus from prior splits at 1676289594058Running coprocessor post-open hooks at 1676289594065 (+7 ms)Region opened successfully at 1676289594065 2023-02-13 11:59:54,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:54,068 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., pid=101, masterSystemTime=1676289594002 2023-02-13 11:59:54,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=97 2023-02-13 11:59:54,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=97, state=SUCCESS; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,34233,1676289577518 in 214 msec 2023-02-13 11:59:54,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:54,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:54,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:54,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 11:59:54,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:54,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:54,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:54,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:54,072 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:54,073 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:54,073 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289594073"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289594073"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289594073"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289594073"}]},"ts":"1676289594073"} 2023-02-13 11:59:54,075 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN in 394 msec 2023-02-13 11:59:54,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:54,077 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:54,077 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:54,078 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 56eee98ead755df3cb16e7153d171fbf columnFamilyName cf 2023-02-13 11:59:54,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened e0ee51e8905082f3fc07662881dfd80c; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66547285, jitterRate=-0.008368179202079773}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:54,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for e0ee51e8905082f3fc07662881dfd80c: Running coprocessor pre-open hook at 1676289594009Writing region info on filesystem at 1676289594009Initializing all the Stores at 1676289594011 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289594011Cleaning up temporary data from old regions at 1676289594066 (+55 ms)Cleaning up detritus from prior splits at 1676289594067 (+1 ms)Running coprocessor post-open hooks at 1676289594079 (+12 ms)Region opened successfully at 1676289594079 2023-02-13 11:59:54,079 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:54,079 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=93 2023-02-13 11:59:54,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., pid=103, masterSystemTime=1676289594003 2023-02-13 11:59:54,079 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=93, state=SUCCESS; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478 in 226 msec 2023-02-13 11:59:54,080 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:54,080 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:54,081 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a40bfc382ab282660f54577d5b7b9575 columnFamilyName cf 2023-02-13 11:59:54,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:54,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:54,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:54,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 11:59:54,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:54,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:54,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:54,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:54,082 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN in 404 msec 2023-02-13 11:59:54,082 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:54,082 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289594082"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289594082"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289594082"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289594082"}]},"ts":"1676289594082"} 2023-02-13 11:59:54,085 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=95 2023-02-13 11:59:54,086 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=95, state=SUCCESS; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,35305,1676289577427 in 230 msec 2023-02-13 11:59:54,088 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN in 409 msec 2023-02-13 11:59:54,093 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:54,095 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:54,095 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:54,096 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b55b9ce69d00ead52d760ba43038e9a columnFamilyName cf 2023-02-13 11:59:54,099 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:54,100 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(310): Store=a40bfc382ab282660f54577d5b7b9575/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:54,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:54,101 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:54,101 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(310): Store=56eee98ead755df3cb16e7153d171fbf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:54,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:54,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:54,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:54,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:54,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:54,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a40bfc382ab282660f54577d5b7b9575; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62922500, jitterRate=-0.06238168478012085}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:54,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a40bfc382ab282660f54577d5b7b9575: Running coprocessor pre-open hook at 1676289594072Writing region info on filesystem at 1676289594072Initializing all the Stores at 1676289594078 (+6 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289594078Cleaning up temporary data from old regions at 1676289594103 (+25 ms)Cleaning up detritus from prior splits at 1676289594105 (+2 ms)Running coprocessor post-open hooks at 1676289594109 (+4 ms)Region opened successfully at 1676289594109 2023-02-13 11:59:54,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 56eee98ead755df3cb16e7153d171fbf; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70515121, jitterRate=0.050757184624671936}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:54,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 56eee98ead755df3cb16e7153d171fbf: Running coprocessor pre-open hook at 1676289594062Writing region info on filesystem at 1676289594062Initializing all the Stores at 1676289594064 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289594064Cleaning up temporary data from old regions at 1676289594105 (+41 ms)Cleaning up detritus from prior splits at 1676289594106 (+1 ms)Running coprocessor post-open hooks at 1676289594110 (+4 ms)Region opened successfully at 1676289594110 2023-02-13 11:59:54,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., pid=99, masterSystemTime=1676289594002 2023-02-13 11:59:54,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., pid=104, masterSystemTime=1676289594008 2023-02-13 11:59:54,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:54,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:54,115 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:54,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:54,116 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289594115"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289594115"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289594115"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289594115"}]},"ts":"1676289594115"} 2023-02-13 11:59:54,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:54,117 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:54,117 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289594117"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289594117"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289594117"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289594117"}]},"ts":"1676289594117"} 2023-02-13 11:59:54,122 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=96 2023-02-13 11:59:54,122 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=96, state=SUCCESS; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,45135,1676289577478 in 274 msec 2023-02-13 11:59:54,124 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=94 2023-02-13 11:59:54,125 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=94, state=SUCCESS; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,34233,1676289577518 in 259 msec 2023-02-13 11:59:54,124 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN in 445 msec 2023-02-13 11:59:54,129 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN in 450 msec 2023-02-13 11:59:54,130 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:54,130 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(310): Store=2b55b9ce69d00ead52d760ba43038e9a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:54,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:54,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:54,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:54,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 2b55b9ce69d00ead52d760ba43038e9a; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62985947, jitterRate=-0.0614362508058548}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:54,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 2b55b9ce69d00ead52d760ba43038e9a: Running coprocessor pre-open hook at 1676289594082Writing region info on filesystem at 1676289594082Initializing all the Stores at 1676289594082Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289594082Cleaning up temporary data from old regions at 1676289594133 (+51 ms)Cleaning up detritus from prior splits at 1676289594134 (+1 ms)Running coprocessor post-open hooks at 1676289594137 (+3 ms)Region opened successfully at 1676289594137 2023-02-13 11:59:54,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., pid=100, masterSystemTime=1676289594003 2023-02-13 11:59:54,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:54,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:54,141 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:54,142 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289594141"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289594141"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289594141"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289594141"}]},"ts":"1676289594141"} 2023-02-13 11:59:54,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=98 2023-02-13 11:59:54,150 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=98, state=SUCCESS; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 299 msec 2023-02-13 11:59:54,152 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=98, resume processing ppid=92 2023-02-13 11:59:54,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN in 473 msec 2023-02-13 11:59:54,154 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289594153"}]},"ts":"1676289594153"} 2023-02-13 11:59:54,156 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLED in hbase:meta 2023-02-13 11:59:54,186 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676289580536' was successfully enabled. 2023-02-13 11:59:54,193 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676289580536 in 529 msec 2023-02-13 11:59:54,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 11:59:54,416 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 92 completed 2023-02-13 11:59:54,416 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(139): Connect 0x71bcdbcf to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:54,501 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49fe38b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:54,505 DEBUG [hconnection-0x5befc622-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:54,511 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42264, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:54,521 DEBUG [hconnection-0x5befc622-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:54,529 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:44286, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:54,845 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 11:59:54,866 DEBUG [hconnection-0x5befc622-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:54,877 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50568, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:55,114 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 11:59:55,361 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 11:59:55,503 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 11:59:55,741 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 11:59:56,573 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 11:59:57,587 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:57,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676289580536 2023-02-13 11:59:57,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:57,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 11:59:57,591 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289597591"}]},"ts":"1676289597591"} 2023-02-13 11:59:57,592 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLING in hbase:meta 2023-02-13 11:59:57,602 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676289580536 to state=DISABLING 2023-02-13 11:59:57,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN}] 2023-02-13 11:59:57,605 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN 2023-02-13 11:59:57,605 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN 2023-02-13 11:59:57,605 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN 2023-02-13 11:59:57,605 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN 2023-02-13 11:59:57,605 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN 2023-02-13 11:59:57,606 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:57,606 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:57,606 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597606"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289597606"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289597606"}]},"ts":"1676289597606"} 2023-02-13 11:59:57,606 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597606"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289597606"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289597606"}]},"ts":"1676289597606"} 2023-02-13 11:59:57,606 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:57,606 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:57,606 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289597606"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289597606"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289597606"}]},"ts":"1676289597606"} 2023-02-13 11:59:57,606 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:57,606 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597606"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289597606"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289597606"}]},"ts":"1676289597606"} 2023-02-13 11:59:57,606 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289597606"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289597606"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289597606"}]},"ts":"1676289597606"} 2023-02-13 11:59:57,607 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=108, state=RUNNABLE; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:57,610 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=107, state=RUNNABLE; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:57,611 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=111, state=RUNNABLE; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:57,612 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:57,612 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN 2023-02-13 11:59:57,613 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=106, state=RUNNABLE; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:57,614 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:57,614 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597614"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289597614"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289597614"}]},"ts":"1676289597614"} 2023-02-13 11:59:57,615 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=110, state=RUNNABLE; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:57,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:57,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:57,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 11:59:57,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 11:59:57,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:57,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:57,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:57,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:57,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 11:59:57,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 11:59:57,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:57,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:57,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:57,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 11:59:57,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:57,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:57,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 11:59:57,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:57,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 11:59:57,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 11:59:57,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:57,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:57,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289597763Running coprocessor pre-close hooks at 1676289597763Disabling compacts and flushes for region at 1676289597763Disabling writes for close at 1676289597768 (+5 ms)Writing region close event to WAL at 1676289597778 (+10 ms)Running coprocessor post-close hooks at 1676289597782 (+4 ms)Closed at 1676289597783 (+1 ms) 2023-02-13 11:59:57,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289597768Running coprocessor pre-close hooks at 1676289597768Disabling compacts and flushes for region at 1676289597768Disabling writes for close at 1676289597769 (+1 ms)Writing region close event to WAL at 1676289597778 (+9 ms)Running coprocessor post-close hooks at 1676289597782 (+4 ms)Closed at 1676289597783 (+1 ms) 2023-02-13 11:59:57,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 11:59:57,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:57,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289597762Running coprocessor pre-close hooks at 1676289597762Disabling compacts and flushes for region at 1676289597762Disabling writes for close at 1676289597768 (+6 ms)Writing region close event to WAL at 1676289597780 (+12 ms)Running coprocessor post-close hooks at 1676289597784 (+4 ms)Closed at 1676289597785 (+1 ms) 2023-02-13 11:59:57,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:57,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:57,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 11:59:57,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:57,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:57,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 11:59:57,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:57,786 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSED 2023-02-13 11:59:57,787 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289597786"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289597786"}]},"ts":"1676289597786"} 2023-02-13 11:59:57,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:57,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:57,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 11:59:57,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:57,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:57,787 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSED 2023-02-13 11:59:57,787 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597787"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289597787"}]},"ts":"1676289597787"} 2023-02-13 11:59:57,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:57,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:57,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 11:59:57,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:57,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 11:59:57,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:57,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:57,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 11:59:57,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:57,789 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSED 2023-02-13 11:59:57,789 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597789"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289597789"}]},"ts":"1676289597789"} 2023-02-13 11:59:57,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 11:59:57,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:57,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289597786Running coprocessor pre-close hooks at 1676289597786Disabling compacts and flushes for region at 1676289597786Disabling writes for close at 1676289597786Writing region close event to WAL at 1676289597786Running coprocessor post-close hooks at 1676289597791 (+5 ms)Closed at 1676289597792 (+1 ms) 2023-02-13 11:59:57,793 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=111 2023-02-13 11:59:57,793 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=111, state=SUCCESS; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 178 msec 2023-02-13 11:59:57,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 11:59:57,794 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-13 11:59:57,794 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:57,794 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,45135,1676289577478 in 178 msec 2023-02-13 11:59:57,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 11:59:57,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN in 190 msec 2023-02-13 11:59:57,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:57,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289597788Running coprocessor pre-close hooks at 1676289597788Disabling compacts and flushes for region at 1676289597788Disabling writes for close at 1676289597788Writing region close event to WAL at 1676289597789 (+1 ms)Running coprocessor post-close hooks at 1676289597795 (+6 ms)Closed at 1676289597795 2023-02-13 11:59:57,796 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSED 2023-02-13 11:59:57,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:57,797 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597796"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289597796"}]},"ts":"1676289597796"} 2023-02-13 11:59:57,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289597787Running coprocessor pre-close hooks at 1676289597787Disabling compacts and flushes for region at 1676289597787Disabling writes for close at 1676289597788 (+1 ms)Writing region close event to WAL at 1676289597789 (+1 ms)Running coprocessor post-close hooks at 1676289597796 (+7 ms)Closed at 1676289597797 (+1 ms) 2023-02-13 11:59:57,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=107 2023-02-13 11:59:57,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN in 191 msec 2023-02-13 11:59:57,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=107, state=SUCCESS; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,34233,1676289577518 in 185 msec 2023-02-13 11:59:57,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:57,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN in 195 msec 2023-02-13 11:59:57,801 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSED 2023-02-13 11:59:57,801 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289597801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289597801"}]},"ts":"1676289597801"} 2023-02-13 11:59:57,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:57,801 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSED 2023-02-13 11:59:57,802 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289597801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289597801"}]},"ts":"1676289597801"} 2023-02-13 11:59:57,802 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=108 2023-02-13 11:59:57,802 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=108, state=SUCCESS; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,35305,1676289577427 in 193 msec 2023-02-13 11:59:57,803 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN in 199 msec 2023-02-13 11:59:57,804 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=110 2023-02-13 11:59:57,804 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=110, state=SUCCESS; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,34233,1676289577518 in 187 msec 2023-02-13 11:59:57,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=106 2023-02-13 11:59:57,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=106, state=SUCCESS; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478 in 191 msec 2023-02-13 11:59:57,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN in 201 msec 2023-02-13 11:59:57,805 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-02-13 11:59:57,805 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN in 201 msec 2023-02-13 11:59:57,806 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289597806"}]},"ts":"1676289597806"} 2023-02-13 11:59:57,807 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLED in hbase:meta 2023-02-13 11:59:57,816 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676289580536 to state=DISABLED 2023-02-13 11:59:57,818 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676289580536 in 230 msec 2023-02-13 11:59:57,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 11:59:57,842 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 105 completed 2023-02-13 11:59:57,850 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:57,851 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:57,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289597851 (current time:1676289597851). 2023-02-13 11:59:57,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 11:59:57,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x5383d525 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:57,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43f70f70, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:57,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:57,865 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42278, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:57,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5383d525 to 127.0.0.1:63172 2023-02-13 11:59:57,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:57,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 11:59:57,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:57,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:57,870 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:57,871 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE; timeout=600000, timestamp=1676290197871 2023-02-13 11:59:57,871 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676289580536-1676289597850 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 2023-02-13 11:59:57,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 11:59:57,871 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 11:59:57,871 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 11:59:57,887 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 11:59:57,890 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:57,890 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:57,890 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:57,890 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:57,892 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:57,892 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:57,894 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:57,894 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:57,894 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:57,894 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c] hfiles 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480] hfiles 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:57,895 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23] hfiles 2023-02-13 11:59:57,896 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:57,896 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf] hfiles 2023-02-13 11:59:57,896 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:57,896 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3] hfiles 2023-02-13 11:59:57,896 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:57,896 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0] hfiles 2023-02-13 11:59:57,896 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:57,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352] hfiles 2023-02-13 11:59:57,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 11:59:57,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 11:59:57,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 11:59:57,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 11:59:57,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 11:59:57,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 11:59:57,936 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } as finished. 2023-02-13 11:59:57,936 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:57,937 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:57,959 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676289580536-1676289597850 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:57,979 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676289580536-1676289597850 of table testRestoreSnapshot-1676289580536 completed 2023-02-13 11:59:57,979 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:57,980 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:57,980 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676289580536-1676289597850 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 at 1676289597871Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } at 1676289597887 (+16 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676289580536-1676289597850 at 1676289597936 (+49 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676289580536-1676289597850 at 1676289597936Writing data manifest for hbase-failsafe-emptySnaptb-1676289580536-1676289597850 at 1676289597940 (+4 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676289580536-1676289597850 at 1676289597951 (+11 ms)Snapshot hbase-failsafe-emptySnaptb-1676289580536-1676289597850 of table testRestoreSnapshot-1676289580536 completed at 1676289597979 (+28 ms) 2023-02-13 11:59:57,988 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 11:59:57,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE in 120 msec 2023-02-13 11:59:58,122 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 11:59:58,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 11:59:58,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676289580536-1676289597850 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 11:59:58,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "emptySnaptb-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289582198 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 11:59:58,130 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676289580536 as table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:58,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 11:59:58,146 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000003 2023-02-13 11:59:58,147 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000002 2023-02-13 11:59:58,147 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000003 2023-02-13 11:59:58,148 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 11:59:58,150 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289582198 type: DISABLED version: 2 ttl: 0 2023-02-13 11:59:58,150 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536 2023-02-13 11:59:58,159 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:58,163 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 11:59:58,165 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 11:59:58,166 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 11:59:58,168 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 11:59:58,169 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 11:59:58,170 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 11:59:58,170 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,171 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,171 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,171 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,171 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,171 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,177 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/1e304a838f584997acee189c66a2bea0 2023-02-13 11:59:58,177 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/ff22144d81504ead9754a1bdee26e1cf 2023-02-13 11:59:58,178 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/6f4565b390274a5e9c444537d8b9bca3 2023-02-13 11:59:58,178 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/276801e0d81b49fd9821e25bfaecd480 2023-02-13 11:59:58,179 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/d847b93f4cdb425190d84b39613c4e23 2023-02-13 11:59:58,179 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 11:59:58,180 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289582198 type: DISABLED version: 2 ttl: 0 2023-02-13 11:59:58,181 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289598181"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:58,181 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289598181"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:58,181 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289598181"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:58,181 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289598181"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:58,181 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289598181"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:58,181 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289598181"}]},"ts":"9223372036854775807"} 2023-02-13 11:59:58,184 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 11:59:58,184 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-13 11:59:58,184 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289598182"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289598182"}]},"ts":"1676289598182"} 2023-02-13 11:59:58,184 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598182"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289598182"}]},"ts":"1676289598182"} 2023-02-13 11:59:58,184 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289598182"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289598182"}]},"ts":"1676289598182"} 2023-02-13 11:59:58,184 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598182"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289598182"}]},"ts":"1676289598182"} 2023-02-13 11:59:58,184 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598182"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289598182"}]},"ts":"1676289598182"} 2023-02-13 11:59:58,184 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598182"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289598182"}]},"ts":"1676289598182"} 2023-02-13 11:59:58,192 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 11:59:58,192 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 11:59:58,192 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-13 11:59:58,192 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676289580536 completed! 2023-02-13 11:59:58,194 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "emptySnaptb-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289582198 type: DISABLED version: 2 ttl: 0 ) in 64 msec 2023-02-13 11:59:58,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 11:59:58,385 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676289580536, procId: 119 completed 2023-02-13 11:59:58,386 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:58,388 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "hbase-failsafe-emptySnaptb-1676289580536-1676289597850" 2023-02-13 11:59:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676289580536-1676289597850 2023-02-13 11:59:58,394 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:58,394 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676289580536 2023-02-13 11:59:58,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:58,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 11:59:58,409 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676289580536 2023-02-13 11:59:58,409 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289598409"}]},"ts":"1676289598409"} 2023-02-13 11:59:58,410 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLING in hbase:meta 2023-02-13 11:59:58,442 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 11:59:58,442 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN}] 2023-02-13 11:59:58,444 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN 2023-02-13 11:59:58,444 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN 2023-02-13 11:59:58,444 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN 2023-02-13 11:59:58,444 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN 2023-02-13 11:59:58,444 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:58,444 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:58,444 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN 2023-02-13 11:59:58,444 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:58,444 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:58,445 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN 2023-02-13 11:59:58,445 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:58,446 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 11:59:58,595 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 11:59:58,595 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 11:59:58,595 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 11:59:58,595 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 11:59:58,595 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 11:59:58,601 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:58,601 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:58,601 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289598601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289598601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289598601"}]},"ts":"1676289598601"} 2023-02-13 11:59:58,601 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:58,601 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:58,601 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289598601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289598601"}]},"ts":"1676289598601"} 2023-02-13 11:59:58,602 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289598601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289598601"}]},"ts":"1676289598601"} 2023-02-13 11:59:58,601 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289598601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289598601"}]},"ts":"1676289598601"} 2023-02-13 11:59:58,602 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:58,602 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598602"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289598602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289598602"}]},"ts":"1676289598602"} 2023-02-13 11:59:58,604 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:58,612 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=122, state=RUNNABLE; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:58,613 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=123, state=RUNNABLE; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:58,617 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=125, state=RUNNABLE; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:58,618 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=124, state=RUNNABLE; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:58,621 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:58,622 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289598621"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289598621"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289598621"}]},"ts":"1676289598621"} 2023-02-13 11:59:58,629 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=121, state=RUNNABLE; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:58,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 11:59:58,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 11:59:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,765 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,766 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:58,766 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 11:59:58,767 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ee51e8905082f3fc07662881dfd80c columnFamilyName cf 2023-02-13 11:59:58,769 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(310): Store=e0ee51e8905082f3fc07662881dfd80c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:58,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:58,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:58,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 11:59:58,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 11:59:58,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:58,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:58,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:58,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened e0ee51e8905082f3fc07662881dfd80c; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63162465, jitterRate=-0.0588059276342392}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:58,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for e0ee51e8905082f3fc07662881dfd80c: Running coprocessor pre-open hook at 1676289598764Writing region info on filesystem at 1676289598764Initializing all the Stores at 1676289598764Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289598764Cleaning up temporary data from old regions at 1676289598772 (+8 ms)Cleaning up detritus from prior splits at 1676289598772Running coprocessor post-open hooks at 1676289598778 (+6 ms)Region opened successfully at 1676289598778 2023-02-13 11:59:58,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., pid=129, masterSystemTime=1676289598760 2023-02-13 11:59:58,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:58,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:58,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:58,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 11:59:58,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:58,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,782 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:58,783 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598782"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289598782"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289598782"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289598782"}]},"ts":"1676289598782"} 2023-02-13 11:59:58,786 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=123 2023-02-13 11:59:58,786 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=123, state=SUCCESS; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478 in 171 msec 2023-02-13 11:59:58,788 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN in 344 msec 2023-02-13 11:59:58,792 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,792 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,792 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,793 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:58,793 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 11:59:58,794 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:58,794 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:58,794 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 11:59:58,794 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 11:59:58,794 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c62ea170afdbf159d444aa319925e012 columnFamilyName cf 2023-02-13 11:59:58,795 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b55b9ce69d00ead52d760ba43038e9a columnFamilyName cf 2023-02-13 11:59:58,795 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a40bfc382ab282660f54577d5b7b9575 columnFamilyName cf 2023-02-13 11:59:58,795 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(310): Store=c62ea170afdbf159d444aa319925e012/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:58,795 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(310): Store=2b55b9ce69d00ead52d760ba43038e9a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:58,796 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(310): Store=a40bfc382ab282660f54577d5b7b9575/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:58,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:58,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:58,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:58,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened c62ea170afdbf159d444aa319925e012; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62664520, jitterRate=-0.06622588634490967}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:58,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a40bfc382ab282660f54577d5b7b9575; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63843398, jitterRate=-0.04865923523902893}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:58,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for c62ea170afdbf159d444aa319925e012: Running coprocessor pre-open hook at 1676289598776Writing region info on filesystem at 1676289598776Initializing all the Stores at 1676289598776Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289598777 (+1 ms)Cleaning up temporary data from old regions at 1676289598798 (+21 ms)Cleaning up detritus from prior splits at 1676289598798Running coprocessor post-open hooks at 1676289598800 (+2 ms)Region opened successfully at 1676289598800 2023-02-13 11:59:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a40bfc382ab282660f54577d5b7b9575: Running coprocessor pre-open hook at 1676289598775Writing region info on filesystem at 1676289598775Initializing all the Stores at 1676289598776 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289598776Cleaning up temporary data from old regions at 1676289598798 (+22 ms)Cleaning up detritus from prior splits at 1676289598798Running coprocessor post-open hooks at 1676289598800 (+2 ms)Region opened successfully at 1676289598800 2023-02-13 11:59:58,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 2b55b9ce69d00ead52d760ba43038e9a; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60336662, jitterRate=-0.10091367363929749}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 2b55b9ce69d00ead52d760ba43038e9a: Running coprocessor pre-open hook at 1676289598782Writing region info on filesystem at 1676289598782Initializing all the Stores at 1676289598783 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289598784 (+1 ms)Cleaning up temporary data from old regions at 1676289598798 (+14 ms)Cleaning up detritus from prior splits at 1676289598798Running coprocessor post-open hooks at 1676289598801 (+3 ms)Region opened successfully at 1676289598801 2023-02-13 11:59:58,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., pid=131, masterSystemTime=1676289598772 2023-02-13 11:59:58,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., pid=127, masterSystemTime=1676289598760 2023-02-13 11:59:58,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., pid=130, masterSystemTime=1676289598769 2023-02-13 11:59:58,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:58,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:58,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:58,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 11:59:58,803 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:58,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:58,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:58,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:58,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,804 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:58,804 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289598804"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289598804"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289598804"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289598804"}]},"ts":"1676289598804"} 2023-02-13 11:59:58,804 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598803"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289598803"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289598803"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289598803"}]},"ts":"1676289598803"} 2023-02-13 11:59:58,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:58,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:58,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:58,804 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:58,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 11:59:58,805 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598804"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289598804"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289598804"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289598804"}]},"ts":"1676289598804"} 2023-02-13 11:59:58,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 11:59:58,805 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,808 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,808 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:58,808 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 11:59:58,809 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81f9e5dd49cc8e0565489f1ebf657e0d columnFamilyName cf 2023-02-13 11:59:58,809 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-02-13 11:59:58,809 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,45135,1676289577478 in 202 msec 2023-02-13 11:59:58,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=124 2023-02-13 11:59:58,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=124, state=SUCCESS; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518 in 189 msec 2023-02-13 11:59:58,810 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(310): Store=81f9e5dd49cc8e0565489f1ebf657e0d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:58,810 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:58,810 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 11:59:58,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,811 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 56eee98ead755df3cb16e7153d171fbf columnFamilyName cf 2023-02-13 11:59:58,811 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=125 2023-02-13 11:59:58,811 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=125, state=SUCCESS; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427 in 191 msec 2023-02-13 11:59:58,812 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN in 367 msec 2023-02-13 11:59:58,812 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(310): Store=56eee98ead755df3cb16e7153d171fbf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 11:59:58,812 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN in 368 msec 2023-02-13 11:59:58,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,813 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN in 369 msec 2023-02-13 11:59:58,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:58,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81f9e5dd49cc8e0565489f1ebf657e0d; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73369259, jitterRate=0.0932871550321579}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:58,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Running coprocessor pre-open hook at 1676289598804Writing region info on filesystem at 1676289598804Initializing all the Stores at 1676289598805 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289598805Cleaning up temporary data from old regions at 1676289598813 (+8 ms)Cleaning up detritus from prior splits at 1676289598813Running coprocessor post-open hooks at 1676289598815 (+2 ms)Region opened successfully at 1676289598815 2023-02-13 11:59:58,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:58,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., pid=132, masterSystemTime=1676289598772 2023-02-13 11:59:58,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 56eee98ead755df3cb16e7153d171fbf; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62171022, jitterRate=-0.0735795795917511}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 11:59:58,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 56eee98ead755df3cb16e7153d171fbf: Running coprocessor pre-open hook at 1676289598805Writing region info on filesystem at 1676289598805Initializing all the Stores at 1676289598807 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289598807Cleaning up temporary data from old regions at 1676289598814 (+7 ms)Cleaning up detritus from prior splits at 1676289598814Running coprocessor post-open hooks at 1676289598816 (+2 ms)Region opened successfully at 1676289598816 2023-02-13 11:59:58,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., pid=128, masterSystemTime=1676289598769 2023-02-13 11:59:58,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:58,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:58,818 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:58,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:58,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:58,818 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289598818"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289598818"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289598818"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289598818"}]},"ts":"1676289598818"} 2023-02-13 11:59:58,818 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:58,818 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289598818"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289598818"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289598818"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289598818"}]},"ts":"1676289598818"} 2023-02-13 11:59:58,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=121 2023-02-13 11:59:58,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=121, state=SUCCESS; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,34233,1676289577518 in 191 msec 2023-02-13 11:59:58,821 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=122 2023-02-13 11:59:58,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=122, state=SUCCESS; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427 in 208 msec 2023-02-13 11:59:58,822 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN in 379 msec 2023-02-13 11:59:58,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=120 2023-02-13 11:59:58,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN in 380 msec 2023-02-13 11:59:58,825 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289598824"}]},"ts":"1676289598824"} 2023-02-13 11:59:58,826 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLED in hbase:meta 2023-02-13 11:59:58,887 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676289580536' was successfully enabled. 2023-02-13 11:59:58,889 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676289580536 in 494 msec 2023-02-13 11:59:59,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 11:59:59,161 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 120 completed 2023-02-13 11:59:59,161 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(139): Connect 0x3545c05d to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 11:59:59,178 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24ce7709, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 11:59:59,192 DEBUG [hconnection-0x177317bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:59,198 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:59,205 DEBUG [hconnection-0x177317bc-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:59,209 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50582, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:59,210 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 11:59:59,216 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 11:59:59,223 DEBUG [hconnection-0x177317bc-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 11:59:59,228 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:44290, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 11:59:59,229 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 11:59:59,238 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 11:59:59,241 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 11:59:59,255 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676289580536 2023-02-13 11:59:59,255 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676289580536 2023-02-13 11:59:59,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 11:59:59,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 11:59:59,262 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289599262"}]},"ts":"1676289599262"} 2023-02-13 11:59:59,263 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLING in hbase:meta 2023-02-13 11:59:59,274 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676289580536 to state=DISABLING 2023-02-13 11:59:59,275 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN}] 2023-02-13 11:59:59,277 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN 2023-02-13 11:59:59,278 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:59,278 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289599278"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289599278"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289599278"}]},"ts":"1676289599278"} 2023-02-13 11:59:59,280 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=139, state=RUNNABLE; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:59,283 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN 2023-02-13 11:59:59,285 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN 2023-02-13 11:59:59,285 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:59,285 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN 2023-02-13 11:59:59,285 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289599285"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289599285"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289599285"}]},"ts":"1676289599285"} 2023-02-13 11:59:59,285 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN 2023-02-13 11:59:59,286 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN 2023-02-13 11:59:59,288 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=134, state=RUNNABLE; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:59,288 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:59,288 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599288"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289599288"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289599288"}]},"ts":"1676289599288"} 2023-02-13 11:59:59,289 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 11:59:59,289 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599289"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289599289"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289599289"}]},"ts":"1676289599289"} 2023-02-13 11:59:59,289 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 11:59:59,289 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599289"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289599289"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289599289"}]},"ts":"1676289599289"} 2023-02-13 11:59:59,289 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 11:59:59,289 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599289"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289599289"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289599289"}]},"ts":"1676289599289"} 2023-02-13 11:59:59,290 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=138, state=RUNNABLE; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:59,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=137, state=RUNNABLE; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 11:59:59,292 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=136, state=RUNNABLE; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 11:59:59,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=135, state=RUNNABLE; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 11:59:59,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:59,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 11:59:59,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:59,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:59,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 11:59:59,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:59,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:59,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 11:59:59,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:59,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:59,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 11:59:59,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:59,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:59,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 11:59:59,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:59,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:59,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 11:59:59,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:59,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 11:59:59,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 11:59:59,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 11:59:59,477 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 11:59:59,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289599441Running coprocessor pre-close hooks at 1676289599441Disabling compacts and flushes for region at 1676289599441Disabling writes for close at 1676289599457 (+16 ms)Writing region close event to WAL at 1676289599457Running coprocessor post-close hooks at 1676289599477 (+20 ms)Closed at 1676289599477 2023-02-13 11:59:59,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 11:59:59,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 11:59:59,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289599433Running coprocessor pre-close hooks at 1676289599433Disabling compacts and flushes for region at 1676289599433Disabling writes for close at 1676289599434 (+1 ms)Writing region close event to WAL at 1676289599457 (+23 ms)Running coprocessor post-close hooks at 1676289599477 (+20 ms)Closed at 1676289599510 (+33 ms) 2023-02-13 11:59:59,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289599457Running coprocessor pre-close hooks at 1676289599457Disabling compacts and flushes for region at 1676289599457Disabling writes for close at 1676289599458 (+1 ms)Writing region close event to WAL at 1676289599459 (+1 ms)Running coprocessor post-close hooks at 1676289599510 (+51 ms)Closed at 1676289599510 2023-02-13 11:59:59,512 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71bcdbcf to 127.0.0.1:63172 2023-02-13 11:59:59,512 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:59,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 11:59:59,512 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38e37a0c to 127.0.0.1:63172 2023-02-13 11:59:59,512 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:59,514 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3545c05d to 127.0.0.1:63172 2023-02-13 11:59:59,514 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 11:59:59,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a40bfc382ab282660f54577d5b7b9575 2023-02-13 11:59:59,514 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSED 2023-02-13 11:59:59,515 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599514"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289599514"}]},"ts":"1676289599514"} 2023-02-13 11:59:59,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:59,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 11:59:59,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed e0ee51e8905082f3fc07662881dfd80c 2023-02-13 11:59:59,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:59,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:59,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:59,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 11:59:59,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:59,518 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSED 2023-02-13 11:59:59,518 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599518"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289599518"}]},"ts":"1676289599518"} 2023-02-13 11:59:59,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 56eee98ead755df3cb16e7153d171fbf 2023-02-13 11:59:59,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:59,519 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSED 2023-02-13 11:59:59,519 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599519"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289599519"}]},"ts":"1676289599519"} 2023-02-13 11:59:59,523 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=137 2023-02-13 11:59:59,523 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=137, state=SUCCESS; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518 in 227 msec 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 11:59:59,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 11:59:59,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 11:59:59,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:59,536 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=136 2023-02-13 11:59:59,536 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN in 248 msec 2023-02-13 11:59:59,536 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=136, state=SUCCESS; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478 in 227 msec 2023-02-13 11:59:59,537 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=135 2023-02-13 11:59:59,537 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=135, state=SUCCESS; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427 in 230 msec 2023-02-13 11:59:59,540 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN in 261 msec 2023-02-13 11:59:59,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN in 262 msec 2023-02-13 11:59:59,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 11:59:59,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 11:59:59,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 11:59:59,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 11:59:59,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289599519Running coprocessor pre-close hooks at 1676289599519Disabling compacts and flushes for region at 1676289599519Disabling writes for close at 1676289599533 (+14 ms)Writing region close event to WAL at 1676289599534 (+1 ms)Running coprocessor post-close hooks at 1676289599543 (+9 ms)Closed at 1676289599543 2023-02-13 11:59:59,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 11:59:59,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289599517Running coprocessor pre-close hooks at 1676289599517Disabling compacts and flushes for region at 1676289599517Disabling writes for close at 1676289599533 (+16 ms)Writing region close event to WAL at 1676289599533Running coprocessor post-close hooks at 1676289599544 (+11 ms)Closed at 1676289599544 2023-02-13 11:59:59,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 11:59:59,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289599515Running coprocessor pre-close hooks at 1676289599515Disabling compacts and flushes for region at 1676289599515Disabling writes for close at 1676289599517 (+2 ms)Writing region close event to WAL at 1676289599534 (+17 ms)Running coprocessor post-close hooks at 1676289599544 (+10 ms)Closed at 1676289599544 2023-02-13 11:59:59,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c62ea170afdbf159d444aa319925e012 2023-02-13 11:59:59,547 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSED 2023-02-13 11:59:59,547 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289599547"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289599547"}]},"ts":"1676289599547"} 2023-02-13 11:59:59,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 11:59:59,550 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSED 2023-02-13 11:59:59,550 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289599550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289599550"}]},"ts":"1676289599550"} 2023-02-13 11:59:59,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 11:59:59,554 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSED 2023-02-13 11:59:59,554 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289599554"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289599554"}]},"ts":"1676289599554"} 2023-02-13 11:59:59,557 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=138 2023-02-13 11:59:59,558 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=138, state=SUCCESS; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427 in 263 msec 2023-02-13 11:59:59,565 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=139 2023-02-13 11:59:59,565 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=139, state=SUCCESS; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,45135,1676289577478 in 275 msec 2023-02-13 11:59:59,566 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN in 283 msec 2023-02-13 11:59:59,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=134 2023-02-13 11:59:59,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=134, state=SUCCESS; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,34233,1676289577518 in 269 msec 2023-02-13 11:59:59,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN in 290 msec 2023-02-13 11:59:59,571 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=134, resume processing ppid=133 2023-02-13 11:59:59,572 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN in 291 msec 2023-02-13 11:59:59,572 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289599572"}]},"ts":"1676289599572"} 2023-02-13 11:59:59,575 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLED in hbase:meta 2023-02-13 11:59:59,677 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676289580536 to state=DISABLED 2023-02-13 11:59:59,681 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676289580536 in 422 msec 2023-02-13 12:00:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 12:00:00,015 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 133 completed 2023-02-13 12:00:00,029 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:00,031 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 12:00:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289600031 (current time:1676289600031). 2023-02-13 12:00:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:00:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x5c9771cb to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:00:00,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@259594bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:00:00,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:00,073 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:46396, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:00,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c9771cb to 127.0.0.1:63172 2023-02-13 12:00:00,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:00,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:00:00,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 12:00:00,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 12:00:00,078 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 12:00:00,079 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE; timeout=600000, timestamp=1676290200079 2023-02-13 12:00:00,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } 2023-02-13 12:00:00,079 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676289580536-1676289600029 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,080 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:00:00,080 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:00:00,105 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 12:00:00,111 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:00,111 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:00,111 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:00,111 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:00,111 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:00,112 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:00,111 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:00,112 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:00,112 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:00,112 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:00,112 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:00,112 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:00,113 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:00,113 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:00,115 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:00,115 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:00,115 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:00,115 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:00,118 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:00,123 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:00,124 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:00,276 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } as finished. 2023-02-13 12:00:00,276 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:00,277 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:00,330 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:00:00,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 12:00:00,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 12:00:00,331 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 12:00:00,332 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676289580536-1676289600029 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:00,385 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676289580536-1676289600029 of table testRestoreSnapshot-1676289580536 completed 2023-02-13 12:00:00,385 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:00,385 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:00,385 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676289580536-1676289600029 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676289580536 at 1676289600079Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } at 1676289600105 (+26 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676289580536-1676289600029 at 1676289600276 (+171 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676289580536-1676289600029 at 1676289600277 (+1 ms)Writing data manifest for hbase-failsafe-snaptb1-1676289580536-1676289600029 at 1676289600284 (+7 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676289580536-1676289600029 at 1676289600313 (+29 ms)Snapshot hbase-failsafe-snaptb1-1676289580536-1676289600029 of table testRestoreSnapshot-1676289580536 completed at 1676289600385 (+72 ms) 2023-02-13 12:00:00,387 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE 2023-02-13 12:00:00,389 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676289580536, type=EXCLUSIVE in 311 msec 2023-02-13 12:00:00,832 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:00:00,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 } is done 2023-02-13 12:00:00,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676289580536-1676289600029 table=testRestoreSnapshot-1676289580536 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:00:00,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 12:00:00,849 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676289580536 as table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 12:00:00,865 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000004 2023-02-13 12:00:00,866 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000003 2023-02-13 12:00:00,866 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000004 2023-02-13 12:00:00,867 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } 2023-02-13 12:00:00,870 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 2023-02-13 12:00:00,870 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,878 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,880 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,883 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,886 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,890 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,893 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,897 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,900 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,903 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,908 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,912 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,917 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,922 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef of snapshot=snaptb1-1676289580536 to region=aab21f680c1fff98d3160c527e41e1be table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,927 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:00,928 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:00,928 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:00,928 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:00,928 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:00,928 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:00,933 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d847b93f4cdb425190d84b39613c4e23 of snapshot=snaptb1-1676289580536 to region=81f9e5dd49cc8e0565489f1ebf657e0d table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,933 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 29f3eaf3b2c946079123a6bd68c2201a of snapshot=snaptb1-1676289580536 to region=2b55b9ce69d00ead52d760ba43038e9a table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,934 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 1e304a838f584997acee189c66a2bea0 of snapshot=snaptb1-1676289580536 to region=56eee98ead755df3cb16e7153d171fbf table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,934 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 276801e0d81b49fd9821e25bfaecd480 of snapshot=snaptb1-1676289580536 to region=a40bfc382ab282660f54577d5b7b9575 table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,935 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 599dabece3764fa8a0da81bc49ad9d1c of snapshot=snaptb1-1676289580536 to region=e0ee51e8905082f3fc07662881dfd80c table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,937 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 6f4565b390274a5e9c444537d8b9bca3 of snapshot=snaptb1-1676289580536 to region=c62ea170afdbf159d444aa319925e012 table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,950 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink ff22144d81504ead9754a1bdee26e1cf of snapshot=snaptb1-1676289580536 to region=2b55b9ce69d00ead52d760ba43038e9a table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,956 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink efd9c8e078ce4850b8dcbe3b73c8a105 of snapshot=snaptb1-1676289580536 to region=56eee98ead755df3cb16e7153d171fbf table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,963 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d71da190e77346458dd9bdfc6068675b of snapshot=snaptb1-1676289580536 to region=a40bfc382ab282660f54577d5b7b9575 table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,963 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c193a7455d3d48f29f60b9563d1f1627 of snapshot=snaptb1-1676289580536 to region=e0ee51e8905082f3fc07662881dfd80c table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,964 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a5cf918d60df457f8c1a52e0029a35e4 of snapshot=snaptb1-1676289580536 to region=c62ea170afdbf159d444aa319925e012 table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,963 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink fe54fb5b90734f3bb8a4e7c25ecba3ca of snapshot=snaptb1-1676289580536 to region=81f9e5dd49cc8e0565489f1ebf657e0d table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:00,969 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 2023-02-13 12:00:00,970 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289600970"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:00,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289600970"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:00,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289600970"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:00,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289600970"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:00,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289600970"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:00,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289600970"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:00,973 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:00:00,973 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-13 12:00:00,973 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289600971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289600971"}]},"ts":"1676289600971"} 2023-02-13 12:00:00,973 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289600971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289600971"}]},"ts":"1676289600971"} 2023-02-13 12:00:00,973 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289600971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289600971"}]},"ts":"1676289600971"} 2023-02-13 12:00:00,973 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289600971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289600971"}]},"ts":"1676289600971"} 2023-02-13 12:00:00,973 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289600971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289600971"}]},"ts":"1676289600971"} 2023-02-13 12:00:00,973 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289600971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289600971"}]},"ts":"1676289600971"} 2023-02-13 12:00:00,975 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:00:00,975 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 12:00:00,975 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-13 12:00:00,975 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676289580536 table=testRestoreSnapshot-1676289580536 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676289580536 completed! 2023-02-13 12:00:00,978 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 ) in 129 msec 2023-02-13 12:00:01,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 12:00:01,103 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676289580536, procId: 147 completed 2023-02-13 12:00:01,103 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:01,104 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "hbase-failsafe-snaptb1-1676289580536-1676289600029" 2023-02-13 12:00:01,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676289580536-1676289600029 2023-02-13 12:00:01,107 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676289580536 2023-02-13 12:00:01,108 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676289580536 2023-02-13 12:00:01,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 12:00:01,112 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676289580536 2023-02-13 12:00:01,112 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289601112"}]},"ts":"1676289601112"} 2023-02-13 12:00:01,113 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLING in hbase:meta 2023-02-13 12:00:01,144 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:00:01,145 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN}] 2023-02-13 12:00:01,147 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN 2023-02-13 12:00:01,147 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN 2023-02-13 12:00:01,147 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN 2023-02-13 12:00:01,147 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN 2023-02-13 12:00:01,148 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:00:01,148 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:00:01,148 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:00:01,148 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:00:01,148 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN 2023-02-13 12:00:01,149 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN 2023-02-13 12:00:01,149 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:00:01,150 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:00:01,298 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:00:01,298 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:00:01,298 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:00:01,298 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:00:01,298 DEBUG [jenkins-hbase16:39205] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:00:01,303 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:01,303 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601303"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289601303"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289601303"}]},"ts":"1676289601303"} 2023-02-13 12:00:01,304 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:01,304 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:01,304 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289601304"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289601304"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289601304"}]},"ts":"1676289601304"} 2023-02-13 12:00:01,304 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:01,304 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289601304"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289601304"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289601304"}]},"ts":"1676289601304"} 2023-02-13 12:00:01,304 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:01,304 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601303"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289601303"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289601303"}]},"ts":"1676289601303"} 2023-02-13 12:00:01,307 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601304"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289601304"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289601304"}]},"ts":"1676289601304"} 2023-02-13 12:00:01,307 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=153, state=RUNNABLE; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:01,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=149, state=RUNNABLE; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:01,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=154, state=RUNNABLE; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:01,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=150, state=RUNNABLE; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:01,312 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:01,312 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601312"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289601312"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289601312"}]},"ts":"1676289601312"} 2023-02-13 12:00:01,312 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=152, state=RUNNABLE; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:01,319 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:01,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 12:00:01,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:00:01,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:01,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:01,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:01,467 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:01,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:01,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:00:01,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:01,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:01,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:01,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:01,468 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 12:00:01,468 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 12:00:01,469 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a40bfc382ab282660f54577d5b7b9575 columnFamilyName cf 2023-02-13 12:00:01,476 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:01,476 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:01,478 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 12:00:01,478 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 12:00:01,478 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 12:00:01,478 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 12:00:01,479 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ee51e8905082f3fc07662881dfd80c columnFamilyName cf 2023-02-13 12:00:01,480 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c62ea170afdbf159d444aa319925e012 columnFamilyName cf 2023-02-13 12:00:01,510 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 2023-02-13 12:00:01,517 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b 2023-02-13 12:00:01,517 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(310): Store=a40bfc382ab282660f54577d5b7b9575/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:01,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:01,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:01,524 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 12:00:01,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:01,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a40bfc382ab282660f54577d5b7b9575; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68682630, jitterRate=0.02345094084739685}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:01,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a40bfc382ab282660f54577d5b7b9575: Running coprocessor pre-open hook at 1676289601465Writing region info on filesystem at 1676289601465Initializing all the Stores at 1676289601466 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289601466Cleaning up temporary data from old regions at 1676289601523 (+57 ms)Cleaning up detritus from prior splits at 1676289601523Running coprocessor post-open hooks at 1676289601526 (+3 ms)Region opened successfully at 1676289601526 2023-02-13 12:00:01,531 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 2023-02-13 12:00:01,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., pid=159, masterSystemTime=1676289601461 2023-02-13 12:00:01,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:01,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:01,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:01,536 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:01,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:00:01,536 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601536"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289601536"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289601536"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289601536"}]},"ts":"1676289601536"} 2023-02-13 12:00:01,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:01,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:01,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:01,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:01,538 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 2023-02-13 12:00:01,538 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(310): Store=e0ee51e8905082f3fc07662881dfd80c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:01,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=152 2023-02-13 12:00:01,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=152, state=SUCCESS; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 227 msec 2023-02-13 12:00:01,543 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN in 396 msec 2023-02-13 12:00:01,551 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:01,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:01,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:01,554 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 12:00:01,554 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 12:00:01,555 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81f9e5dd49cc8e0565489f1ebf657e0d columnFamilyName cf 2023-02-13 12:00:01,556 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 12:00:01,556 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(310): Store=c62ea170afdbf159d444aa319925e012/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:01,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:01,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:01,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:01,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened e0ee51e8905082f3fc07662881dfd80c; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75205977, jitterRate=0.12065638601779938}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:01,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for e0ee51e8905082f3fc07662881dfd80c: Running coprocessor pre-open hook at 1676289601468Writing region info on filesystem at 1676289601468Initializing all the Stores at 1676289601469 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289601469Cleaning up temporary data from old regions at 1676289601555 (+86 ms)Cleaning up detritus from prior splits at 1676289601555Running coprocessor post-open hooks at 1676289601559 (+4 ms)Region opened successfully at 1676289601559 2023-02-13 12:00:01,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., pid=160, masterSystemTime=1676289601462 2023-02-13 12:00:01,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:01,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:01,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:01,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:01,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:00:01,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:01,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:01,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened c62ea170afdbf159d444aa319925e012; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69304779, jitterRate=0.032721683382987976}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:01,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for c62ea170afdbf159d444aa319925e012: Running coprocessor pre-open hook at 1676289601465Writing region info on filesystem at 1676289601465Initializing all the Stores at 1676289601466 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289601466Cleaning up temporary data from old regions at 1676289601559 (+93 ms)Cleaning up detritus from prior splits at 1676289601559Running coprocessor post-open hooks at 1676289601562 (+3 ms)Region opened successfully at 1676289601562 2023-02-13 12:00:01,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:01,563 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:01,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:01,563 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601563"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289601563"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289601563"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289601563"}]},"ts":"1676289601563"} 2023-02-13 12:00:01,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., pid=155, masterSystemTime=1676289601461 2023-02-13 12:00:01,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:01,567 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:01,567 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:01,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:00:01,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:01,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:01,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:01,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:01,568 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:01,568 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601568"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289601568"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289601568"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289601568"}]},"ts":"1676289601568"} 2023-02-13 12:00:01,570 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:01,570 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:01,572 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 12:00:01,572 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 12:00:01,573 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b55b9ce69d00ead52d760ba43038e9a columnFamilyName cf 2023-02-13 12:00:01,573 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=153 2023-02-13 12:00:01,573 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-13 12:00:01,580 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=153, state=SUCCESS; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 264 msec 2023-02-13 12:00:01,580 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 251 msec 2023-02-13 12:00:01,583 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 12:00:01,583 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 12:00:01,583 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 56eee98ead755df3cb16e7153d171fbf columnFamilyName cf 2023-02-13 12:00:01,584 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 2023-02-13 12:00:01,588 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN in 435 msec 2023-02-13 12:00:01,588 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN in 435 msec 2023-02-13 12:00:01,603 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 12:00:01,604 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(310): Store=81f9e5dd49cc8e0565489f1ebf657e0d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:01,604 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 2023-02-13 12:00:01,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:01,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:01,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:01,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81f9e5dd49cc8e0565489f1ebf657e0d; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71972962, jitterRate=0.072480708360672}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:01,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Running coprocessor pre-open hook at 1676289601537Writing region info on filesystem at 1676289601537Initializing all the Stores at 1676289601538 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289601538Cleaning up temporary data from old regions at 1676289601612 (+74 ms)Cleaning up detritus from prior splits at 1676289601612Running coprocessor post-open hooks at 1676289601616 (+4 ms)Region opened successfully at 1676289601616 2023-02-13 12:00:01,617 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., pid=156, masterSystemTime=1676289601461 2023-02-13 12:00:01,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:01,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:01,619 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:01,619 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289601619"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289601619"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289601619"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289601619"}]},"ts":"1676289601619"} 2023-02-13 12:00:01,625 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=149 2023-02-13 12:00:01,625 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=149, state=SUCCESS; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 314 msec 2023-02-13 12:00:01,626 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 12:00:01,627 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN in 480 msec 2023-02-13 12:00:01,627 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 12:00:01,627 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(310): Store=56eee98ead755df3cb16e7153d171fbf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:01,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:01,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:01,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:01,638 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf 2023-02-13 12:00:01,638 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(310): Store=2b55b9ce69d00ead52d760ba43038e9a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:01,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 56eee98ead755df3cb16e7153d171fbf; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59403716, jitterRate=-0.11481565237045288}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:01,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 56eee98ead755df3cb16e7153d171fbf: Running coprocessor pre-open hook at 1676289601568Writing region info on filesystem at 1676289601568Initializing all the Stores at 1676289601569 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289601569Cleaning up temporary data from old regions at 1676289601631 (+62 ms)Cleaning up detritus from prior splits at 1676289601632 (+1 ms)Running coprocessor post-open hooks at 1676289601639 (+7 ms)Region opened successfully at 1676289601639 2023-02-13 12:00:01,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:01,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., pid=158, masterSystemTime=1676289601461 2023-02-13 12:00:01,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:01,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:01,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:01,643 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:01,643 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289601643"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289601643"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289601643"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289601643"}]},"ts":"1676289601643"} 2023-02-13 12:00:01,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:01,646 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 2b55b9ce69d00ead52d760ba43038e9a; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66693461, jitterRate=-0.006189987063407898}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:01,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 2b55b9ce69d00ead52d760ba43038e9a: Running coprocessor pre-open hook at 1676289601563Writing region info on filesystem at 1676289601563Initializing all the Stores at 1676289601564 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289601564Cleaning up temporary data from old regions at 1676289601642 (+78 ms)Cleaning up detritus from prior splits at 1676289601642Running coprocessor post-open hooks at 1676289601646 (+4 ms)Region opened successfully at 1676289601646 2023-02-13 12:00:01,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., pid=157, masterSystemTime=1676289601462 2023-02-13 12:00:01,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:01,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:01,650 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:01,651 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289601649"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289601649"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289601649"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289601649"}]},"ts":"1676289601649"} 2023-02-13 12:00:01,657 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=150 2023-02-13 12:00:01,657 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=150, state=SUCCESS; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 341 msec 2023-02-13 12:00:01,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN in 512 msec 2023-02-13 12:00:01,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=154 2023-02-13 12:00:01,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=154, state=SUCCESS; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 350 msec 2023-02-13 12:00:01,664 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=154, resume processing ppid=148 2023-02-13 12:00:01,664 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN in 517 msec 2023-02-13 12:00:01,664 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289601664"}]},"ts":"1676289601664"} 2023-02-13 12:00:01,666 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLED in hbase:meta 2023-02-13 12:00:01,699 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676289580536' was successfully enabled. 2023-02-13 12:00:01,702 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676289580536 in 592 msec 2023-02-13 12:00:01,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 12:00:01,863 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 148 completed 2023-02-13 12:00:01,864 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(139): Connect 0x546668f0 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:00:01,883 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@192e11ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:00:01,888 DEBUG [hconnection-0x5ee4869b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:01,892 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:46400, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:01,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:01,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,131 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:00:02,133 DEBUG [hconnection-0x5ee4869b-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:02,134 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:41182, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,336 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:00:02,339 DEBUG [hconnection-0x5ee4869b-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:02,341 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37384, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:02,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,585 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:00:02,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,776 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:00:02,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:02,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,133 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:00:03,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,286 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:00:03,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:03,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:04,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:05,474 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676289580536 2023-02-13 12:00:05,474 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676289580536 2023-02-13 12:00:05,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:05,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 12:00:05,479 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289605478"}]},"ts":"1676289605478"} 2023-02-13 12:00:05,480 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLING in hbase:meta 2023-02-13 12:00:05,491 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676289580536 to state=DISABLING 2023-02-13 12:00:05,497 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN}] 2023-02-13 12:00:05,499 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN 2023-02-13 12:00:05,499 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN 2023-02-13 12:00:05,500 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN 2023-02-13 12:00:05,500 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN 2023-02-13 12:00:05,500 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN 2023-02-13 12:00:05,501 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:05,501 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:05,501 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605501"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289605501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289605501"}]},"ts":"1676289605501"} 2023-02-13 12:00:05,501 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605501"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289605501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289605501"}]},"ts":"1676289605501"} 2023-02-13 12:00:05,501 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:05,501 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289605501"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289605501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289605501"}]},"ts":"1676289605501"} 2023-02-13 12:00:05,501 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:05,501 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605501"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289605501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289605501"}]},"ts":"1676289605501"} 2023-02-13 12:00:05,502 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:05,502 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289605502"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289605502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289605502"}]},"ts":"1676289605502"} 2023-02-13 12:00:05,503 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=165, state=RUNNABLE; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:05,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=163, state=RUNNABLE; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:05,505 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=167, state=RUNNABLE; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:05,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=164, state=RUNNABLE; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:05,508 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=162, state=RUNNABLE; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:05,509 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN 2023-02-13 12:00:05,511 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:05,511 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605511"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289605511"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289605511"}]},"ts":"1676289605511"} 2023-02-13 12:00:05,515 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:05,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:05,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 12:00:05,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:05,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:05,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 12:00:05,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:05,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:05,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 12:00:05,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 12:00:05,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 12:00:05,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:05,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 12:00:05,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:00:05,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:00:05,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:05,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289605660Running coprocessor pre-close hooks at 1676289605660Disabling compacts and flushes for region at 1676289605660Disabling writes for close at 1676289605678 (+18 ms)Writing region close event to WAL at 1676289605732 (+54 ms)Running coprocessor post-close hooks at 1676289605741 (+9 ms)Closed at 1676289605753 (+12 ms) 2023-02-13 12:00:05,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:00:05,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:05,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289605657Running coprocessor pre-close hooks at 1676289605657Disabling compacts and flushes for region at 1676289605657Disabling writes for close at 1676289605659 (+2 ms)Writing region close event to WAL at 1676289605720 (+61 ms)Running coprocessor post-close hooks at 1676289605755 (+35 ms)Closed at 1676289605756 (+1 ms) 2023-02-13 12:00:05,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:05,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:05,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 12:00:05,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:05,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:05,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:05,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 12:00:05,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289605664Running coprocessor pre-close hooks at 1676289605664Disabling compacts and flushes for region at 1676289605664Disabling writes for close at 1676289605678 (+14 ms)Writing region close event to WAL at 1676289605730 (+52 ms)Running coprocessor post-close hooks at 1676289605758 (+28 ms)Closed at 1676289605759 (+1 ms) 2023-02-13 12:00:05,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:05,760 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSED 2023-02-13 12:00:05,760 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289605760"}]},"ts":"1676289605760"} 2023-02-13 12:00:05,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:05,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:05,761 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSED 2023-02-13 12:00:05,762 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289605761"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289605761"}]},"ts":"1676289605761"} 2023-02-13 12:00:05,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:05,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:05,762 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSED 2023-02-13 12:00:05,762 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605762"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289605762"}]},"ts":"1676289605762"} 2023-02-13 12:00:05,765 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-13 12:00:05,765 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,45135,1676289577478 in 247 msec 2023-02-13 12:00:05,768 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=162 2023-02-13 12:00:05,768 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN in 269 msec 2023-02-13 12:00:05,768 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=162, state=SUCCESS; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,35305,1676289577427 in 256 msec 2023-02-13 12:00:05,768 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=164 2023-02-13 12:00:05,768 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=164, state=SUCCESS; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,34233,1676289577518 in 258 msec 2023-02-13 12:00:05,769 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN in 271 msec 2023-02-13 12:00:05,769 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN in 271 msec 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 12:00:05,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 12:00:05,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 12:00:05,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:05,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:00:05,811 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:00:05,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:05,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:05,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289605760Running coprocessor pre-close hooks at 1676289605760Disabling compacts and flushes for region at 1676289605760Disabling writes for close at 1676289605773 (+13 ms)Writing region close event to WAL at 1676289605798 (+25 ms)Running coprocessor post-close hooks at 1676289605811 (+13 ms)Closed at 1676289605812 (+1 ms) 2023-02-13 12:00:05,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289605757Running coprocessor pre-close hooks at 1676289605757Disabling compacts and flushes for region at 1676289605757Disabling writes for close at 1676289605759 (+2 ms)Writing region close event to WAL at 1676289605791 (+32 ms)Running coprocessor post-close hooks at 1676289605812 (+21 ms)Closed at 1676289605813 (+1 ms) 2023-02-13 12:00:05,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:00:05,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:05,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289605762Running coprocessor pre-close hooks at 1676289605762Disabling compacts and flushes for region at 1676289605762Disabling writes for close at 1676289605773 (+11 ms)Writing region close event to WAL at 1676289605802 (+29 ms)Running coprocessor post-close hooks at 1676289605814 (+12 ms)Closed at 1676289605815 (+1 ms) 2023-02-13 12:00:05,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:05,816 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSED 2023-02-13 12:00:05,816 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605816"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289605816"}]},"ts":"1676289605816"} 2023-02-13 12:00:05,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:05,817 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSED 2023-02-13 12:00:05,817 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289605817"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289605817"}]},"ts":"1676289605817"} 2023-02-13 12:00:05,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:05,822 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSED 2023-02-13 12:00:05,822 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289605822"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289605822"}]},"ts":"1676289605822"} 2023-02-13 12:00:05,824 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=163 2023-02-13 12:00:05,825 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=163, state=SUCCESS; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,45135,1676289577478 in 317 msec 2023-02-13 12:00:05,826 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=167 2023-02-13 12:00:05,826 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=167, state=SUCCESS; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 318 msec 2023-02-13 12:00:05,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=165 2023-02-13 12:00:05,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=165, state=SUCCESS; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,35305,1676289577427 in 319 msec 2023-02-13 12:00:05,826 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN in 328 msec 2023-02-13 12:00:05,827 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN in 329 msec 2023-02-13 12:00:05,827 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=165, resume processing ppid=161 2023-02-13 12:00:05,827 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN in 329 msec 2023-02-13 12:00:05,828 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289605828"}]},"ts":"1676289605828"} 2023-02-13 12:00:05,829 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLED in hbase:meta 2023-02-13 12:00:05,835 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676289580536 to state=DISABLED 2023-02-13 12:00:05,837 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676289580536 in 362 msec 2023-02-13 12:00:06,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 12:00:06,238 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 161 completed 2023-02-13 12:00:06,243 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$5(2215): Client=jenkins//188.40.62.62 delete testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,250 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,252 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 12:00:06,259 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:06,259 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:06,259 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:06,259 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:06,259 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:06,259 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:06,261 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits] 2023-02-13 12:00:06,261 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits] 2023-02-13 12:00:06,261 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits] 2023-02-13 12:00:06,261 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits] 2023-02-13 12:00:06,261 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits] 2023-02-13 12:00:06,261 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits] 2023-02-13 12:00:06,265 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 2023-02-13 12:00:06,265 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 12:00:06,265 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 2023-02-13 12:00:06,265 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 2023-02-13 12:00:06,266 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 12:00:06,266 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 2023-02-13 12:00:06,267 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 12:00:06,267 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 12:00:06,267 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 2023-02-13 12:00:06,267 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b 2023-02-13 12:00:06,268 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf 2023-02-13 12:00:06,268 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 12:00:06,271 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/27.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/27.seqid 2023-02-13 12:00:06,271 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/27.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/27.seqid 2023-02-13 12:00:06,272 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/27.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/27.seqid 2023-02-13 12:00:06,271 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/27.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/27.seqid 2023-02-13 12:00:06,272 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:06,272 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/27.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/27.seqid 2023-02-13 12:00:06,272 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:06,272 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:06,272 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:06,273 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:06,273 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/27.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/27.seqid 2023-02-13 12:00:06,273 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:06,273 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676289580536 regions 2023-02-13 12:00:06,274 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be 2023-02-13 12:00:06,274 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf] 2023-02-13 12:00:06,276 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 12:00:06,277 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 2023-02-13 12:00:06,278 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea 2023-02-13 12:00:06,279 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 12:00:06,280 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 2023-02-13 12:00:06,281 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 12:00:06,281 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d 2023-02-13 12:00:06,282 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 12:00:06,283 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 2023-02-13 12:00:06,284 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 12:00:06,285 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 12:00:06,285 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef 2023-02-13 12:00:06,286 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be 2023-02-13 12:00:06,287 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,314 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676289580536 from hbase:meta 2023-02-13 12:00:06,317 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676289580536' descriptor. 2023-02-13 12:00:06,318 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,318 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676289580536' from region states. 2023-02-13 12:00:06,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289606318"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:06,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289606318"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:06,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289606318"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:06,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289606318"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:06,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289606318"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:06,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289606318"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:06,321 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:00:06,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 12:00:06,321 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676289580536' as deleted. 2023-02-13 12:00:06,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676289606321"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:06,323 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676289580536 state from META 2023-02-13 12:00:06,333 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 in 89 msec 2023-02-13 12:00:06,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 12:00:06,504 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 174 completed 2023-02-13 12:00:06,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 12:00:06,538 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676289580536 as table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 12:00:06,552 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/.tabledesc/.tableinfo.0000000001 2023-02-13 12:00:06,554 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 2023-02-13 12:00:06,554 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,554 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,556 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,558 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,560 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,562 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,563 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,565 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,566 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,568 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,569 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,571 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,572 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,573 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(239): region to add: 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=2b55b9ce69d00ead52d760ba43038e9a as 2b55b9ce69d00ead52d760ba43038e9a in snapshot snaptb1-1676289580536 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=c62ea170afdbf159d444aa319925e012 as c62ea170afdbf159d444aa319925e012 in snapshot snaptb1-1676289580536 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=e0ee51e8905082f3fc07662881dfd80c as e0ee51e8905082f3fc07662881dfd80c in snapshot snaptb1-1676289580536 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=a40bfc382ab282660f54577d5b7b9575 as a40bfc382ab282660f54577d5b7b9575 in snapshot snaptb1-1676289580536 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=56eee98ead755df3cb16e7153d171fbf as 56eee98ead755df3cb16e7153d171fbf in snapshot snaptb1-1676289580536 2023-02-13 12:00:06,574 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(585): clone region=81f9e5dd49cc8e0565489f1ebf657e0d as 81f9e5dd49cc8e0565489f1ebf657e0d in snapshot snaptb1-1676289580536 2023-02-13 12:00:06,575 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:06,576 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:06,576 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:06,576 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:06,576 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:06,576 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676289580536', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:06,599 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:06,600 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 276801e0d81b49fd9821e25bfaecd480 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,600 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:06,600 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d847b93f4cdb425190d84b39613c4e23 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,605 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:06,605 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 6f4565b390274a5e9c444537d8b9bca3 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,608 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:06,608 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 29f3eaf3b2c946079123a6bd68c2201a from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,608 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink fe54fb5b90734f3bb8a4e7c25ecba3ca from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,609 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d71da190e77346458dd9bdfc6068675b from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,611 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a5cf918d60df457f8c1a52e0029a35e4 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,614 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:06,614 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 1e304a838f584997acee189c66a2bea0 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,625 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ff22144d81504ead9754a1bdee26e1cf from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,625 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 12:00:06,625 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:06,625 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:06,625 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 12:00:06,625 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:06,625 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:06,625 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289606625Disabling compacts and flushes for region at 1676289606625Disabling writes for close at 1676289606625Writing region close event to WAL at 1676289606625Closed at 1676289606625 2023-02-13 12:00:06,628 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 12:00:06,628 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:06,628 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:06,628 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 12:00:06,628 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:06,628 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:06,628 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289606628Disabling compacts and flushes for region at 1676289606628Disabling writes for close at 1676289606628Writing region close event to WAL at 1676289606628Closed at 1676289606628 2023-02-13 12:00:06,629 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink efd9c8e078ce4850b8dcbe3b73c8a105 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 12:00:06,629 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:06,629 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289606629Disabling compacts and flushes for region at 1676289606629Disabling writes for close at 1676289606629Writing region close event to WAL at 1676289606629Closed at 1676289606629 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 12:00:06,629 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:06,629 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:06,629 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289606629Disabling compacts and flushes for region at 1676289606629Disabling writes for close at 1676289606629Writing region close event to WAL at 1676289606629Closed at 1676289606629 2023-02-13 12:00:06,631 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 12:00:06,631 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:06,631 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:06,632 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 12:00:06,632 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:06,632 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:06,632 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289606631Disabling compacts and flushes for region at 1676289606631Disabling writes for close at 1676289606632 (+1 ms)Writing region close event to WAL at 1676289606632Closed at 1676289606632 2023-02-13 12:00:06,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 12:00:07,005 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:07,005 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 599dabece3764fa8a0da81bc49ad9d1c from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:07,007 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c193a7455d3d48f29f60b9563d1f1627 from cloned region in snapshot snaptb1-1676289580536 to table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:07,010 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 12:00:07,010 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:07,010 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:07,010 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 12:00:07,010 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:07,010 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:07,010 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289607010Disabling compacts and flushes for region at 1676289607010Disabling writes for close at 1676289607010Writing region close event to WAL at 1676289607010Closed at 1676289607010 2023-02-13 12:00:07,010 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 2023-02-13 12:00:07,010 INFO [PEWorker-2] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676289580536 on table=testRestoreSnapshot-1676289580536 completed! 2023-02-13 12:00:07,014 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289607014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289607014"}]},"ts":"1676289607014"} 2023-02-13 12:00:07,014 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289607014"}]},"ts":"1676289607014"} 2023-02-13 12:00:07,015 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289607014"}]},"ts":"1676289607014"} 2023-02-13 12:00:07,015 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289607014"}]},"ts":"1676289607014"} 2023-02-13 12:00:07,015 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289607014"}]},"ts":"1676289607014"} 2023-02-13 12:00:07,015 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289607014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289607014"}]},"ts":"1676289607014"} 2023-02-13 12:00:07,017 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:00:07,018 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289607018"}]},"ts":"1676289607018"} 2023-02-13 12:00:07,019 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLING in hbase:meta 2023-02-13 12:00:07,037 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:00:07,037 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:00:07,037 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:00:07,037 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:00:07,037 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:00:07,037 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN}] 2023-02-13 12:00:07,039 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN 2023-02-13 12:00:07,039 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN 2023-02-13 12:00:07,039 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN 2023-02-13 12:00:07,040 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN 2023-02-13 12:00:07,040 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=false 2023-02-13 12:00:07,040 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN 2023-02-13 12:00:07,040 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=false 2023-02-13 12:00:07,040 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=false 2023-02-13 12:00:07,040 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=false 2023-02-13 12:00:07,040 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=false 2023-02-13 12:00:07,041 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN 2023-02-13 12:00:07,041 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=false 2023-02-13 12:00:07,190 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:00:07,194 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:07,194 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:07,194 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289607194"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289607194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289607194"}]},"ts":"1676289607194"} 2023-02-13 12:00:07,194 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607194"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289607194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289607194"}]},"ts":"1676289607194"} 2023-02-13 12:00:07,194 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:07,194 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:07,194 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289607194"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289607194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289607194"}]},"ts":"1676289607194"} 2023-02-13 12:00:07,194 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607194"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289607194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289607194"}]},"ts":"1676289607194"} 2023-02-13 12:00:07,194 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:07,195 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607194"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289607194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289607194"}]},"ts":"1676289607194"} 2023-02-13 12:00:07,195 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=176, state=RUNNABLE; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:07,198 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=180, state=RUNNABLE; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:07,199 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=181, state=RUNNABLE; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:07,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=177, state=RUNNABLE; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:07,200 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=178, state=RUNNABLE; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:07,200 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:07,200 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607200"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289607200"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289607200"}]},"ts":"1676289607200"} 2023-02-13 12:00:07,202 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=179, state=RUNNABLE; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:07,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 12:00:07,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:07,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:00:07,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:07,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:07,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:07,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:07,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:07,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:00:07,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:07,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:07,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:07,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:07,355 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:07,356 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:07,356 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 12:00:07,356 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf 2023-02-13 12:00:07,357 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a40bfc382ab282660f54577d5b7b9575 columnFamilyName cf 2023-02-13 12:00:07,357 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 12:00:07,357 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf 2023-02-13 12:00:07,357 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0ee51e8905082f3fc07662881dfd80c columnFamilyName cf 2023-02-13 12:00:07,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:07,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:00:07,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:07,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:07,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:07,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:07,371 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:07,372 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 12:00:07,372 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf 2023-02-13 12:00:07,373 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 56eee98ead755df3cb16e7153d171fbf columnFamilyName cf 2023-02-13 12:00:07,376 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 12:00:07,385 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 2023-02-13 12:00:07,385 DEBUG [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 2023-02-13 12:00:07,386 INFO [StoreOpener-e0ee51e8905082f3fc07662881dfd80c-1] regionserver.HStore(310): Store=e0ee51e8905082f3fc07662881dfd80c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:07,387 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 2023-02-13 12:00:07,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:07,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:07,390 DEBUG [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b 2023-02-13 12:00:07,391 INFO [StoreOpener-a40bfc382ab282660f54577d5b7b9575-1] regionserver.HStore(310): Store=a40bfc382ab282660f54577d5b7b9575/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:07,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:07,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:07,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:07,393 DEBUG [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 12:00:07,393 INFO [StoreOpener-56eee98ead755df3cb16e7153d171fbf-1] regionserver.HStore(310): Store=56eee98ead755df3cb16e7153d171fbf/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:07,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:07,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:07,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:07,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:00:07,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened e0ee51e8905082f3fc07662881dfd80c; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60887465, jitterRate=-0.09270606935024261}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:07,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for e0ee51e8905082f3fc07662881dfd80c: Running coprocessor pre-open hook at 1676289607353Writing region info on filesystem at 1676289607353Initializing all the Stores at 1676289607355 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289607355Cleaning up temporary data from old regions at 1676289607388 (+33 ms)Cleaning up detritus from prior splits at 1676289607389 (+1 ms)Running coprocessor post-open hooks at 1676289607396 (+7 ms)Region opened successfully at 1676289607396 2023-02-13 12:00:07,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:07,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c., pid=186, masterSystemTime=1676289607349 2023-02-13 12:00:07,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:07,398 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:07,398 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:07,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:00:07,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:07,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:07,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:07,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:07,404 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:07,404 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607404"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289607404"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289607404"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289607404"}]},"ts":"1676289607404"} 2023-02-13 12:00:07,407 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=178 2023-02-13 12:00:07,408 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=178, state=SUCCESS; OpenRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478 in 206 msec 2023-02-13 12:00:07,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, ASSIGN in 370 msec 2023-02-13 12:00:07,416 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:00:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:00:07,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a40bfc382ab282660f54577d5b7b9575; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59148847, jitterRate=-0.11861349642276764}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:07,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 56eee98ead755df3cb16e7153d171fbf; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66839829, jitterRate=-0.004008933901786804}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:07,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a40bfc382ab282660f54577d5b7b9575: Running coprocessor pre-open hook at 1676289607354Writing region info on filesystem at 1676289607354Initializing all the Stores at 1676289607354Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289607354Cleaning up temporary data from old regions at 1676289607393 (+39 ms)Cleaning up detritus from prior splits at 1676289607393Running coprocessor post-open hooks at 1676289607417 (+24 ms)Region opened successfully at 1676289607417 2023-02-13 12:00:07,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 56eee98ead755df3cb16e7153d171fbf: Running coprocessor pre-open hook at 1676289607359Writing region info on filesystem at 1676289607359Initializing all the Stores at 1676289607360 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289607360Cleaning up temporary data from old regions at 1676289607395 (+35 ms)Cleaning up detritus from prior splits at 1676289607395Running coprocessor post-open hooks at 1676289607418 (+23 ms)Region opened successfully at 1676289607418 2023-02-13 12:00:07,418 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 12:00:07,418 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf 2023-02-13 12:00:07,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf., pid=185, masterSystemTime=1676289607349 2023-02-13 12:00:07,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575., pid=187, masterSystemTime=1676289607350 2023-02-13 12:00:07,419 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81f9e5dd49cc8e0565489f1ebf657e0d columnFamilyName cf 2023-02-13 12:00:07,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:07,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:07,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:07,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:00:07,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:07,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:07,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:07,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:07,422 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:07,422 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607422"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289607422"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289607422"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289607422"}]},"ts":"1676289607422"} 2023-02-13 12:00:07,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:07,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:07,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:07,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:00:07,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676289580536 c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:07,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:07,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:07,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:07,425 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:07,425 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607423"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289607423"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289607423"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289607423"}]},"ts":"1676289607423"} 2023-02-13 12:00:07,429 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=179 2023-02-13 12:00:07,429 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=179, state=SUCCESS; OpenRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518 in 223 msec 2023-02-13 12:00:07,431 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=177 2023-02-13 12:00:07,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, ASSIGN in 392 msec 2023-02-13 12:00:07,431 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=177, state=SUCCESS; OpenRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427 in 228 msec 2023-02-13 12:00:07,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, ASSIGN in 394 msec 2023-02-13 12:00:07,436 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:07,437 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:07,438 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 12:00:07,438 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf 2023-02-13 12:00:07,438 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c62ea170afdbf159d444aa319925e012 columnFamilyName cf 2023-02-13 12:00:07,438 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 12:00:07,438 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf 2023-02-13 12:00:07,439 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b55b9ce69d00ead52d760ba43038e9a columnFamilyName cf 2023-02-13 12:00:07,467 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 12:00:07,468 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 2023-02-13 12:00:07,472 DEBUG [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf 2023-02-13 12:00:07,476 INFO [StoreOpener-2b55b9ce69d00ead52d760ba43038e9a-1] regionserver.HStore(310): Store=2b55b9ce69d00ead52d760ba43038e9a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:07,476 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 2023-02-13 12:00:07,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:07,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:07,479 DEBUG [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 12:00:07,479 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:07,484 INFO [StoreOpener-81f9e5dd49cc8e0565489f1ebf657e0d-1] regionserver.HStore(310): Store=81f9e5dd49cc8e0565489f1ebf657e0d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:07,484 DEBUG [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 12:00:07,488 INFO [StoreOpener-c62ea170afdbf159d444aa319925e012-1] regionserver.HStore(310): Store=c62ea170afdbf159d444aa319925e012/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:07,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:00:07,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:07,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:07,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 2b55b9ce69d00ead52d760ba43038e9a; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66660027, jitterRate=-0.0066881924867630005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:07,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 2b55b9ce69d00ead52d760ba43038e9a: Running coprocessor pre-open hook at 1676289607421Writing region info on filesystem at 1676289607421Initializing all the Stores at 1676289607425 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289607425Cleaning up temporary data from old regions at 1676289607478 (+53 ms)Cleaning up detritus from prior splits at 1676289607478Running coprocessor post-open hooks at 1676289607489 (+11 ms)Region opened successfully at 1676289607489 2023-02-13 12:00:07,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:07,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:07,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a., pid=184, masterSystemTime=1676289607350 2023-02-13 12:00:07,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:07,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:07,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:07,492 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:07,492 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289607492"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289607492"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289607492"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289607492"}]},"ts":"1676289607492"} 2023-02-13 12:00:07,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:07,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=181 2023-02-13 12:00:07,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=181, state=SUCCESS; OpenRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 294 msec 2023-02-13 12:00:07,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:00:07,496 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, ASSIGN in 458 msec 2023-02-13 12:00:07,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81f9e5dd49cc8e0565489f1ebf657e0d; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61436410, jitterRate=-0.08452615141868591}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:07,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Running coprocessor pre-open hook at 1676289607399Writing region info on filesystem at 1676289607399Initializing all the Stores at 1676289607400 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289607400Cleaning up temporary data from old regions at 1676289607489 (+89 ms)Cleaning up detritus from prior splits at 1676289607490 (+1 ms)Running coprocessor post-open hooks at 1676289607498 (+8 ms)Region opened successfully at 1676289607498 2023-02-13 12:00:07,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:00:07,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d., pid=182, masterSystemTime=1676289607349 2023-02-13 12:00:07,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened c62ea170afdbf159d444aa319925e012; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68484571, jitterRate=0.020499631762504578}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:07,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for c62ea170afdbf159d444aa319925e012: Running coprocessor pre-open hook at 1676289607424Writing region info on filesystem at 1676289607424Initializing all the Stores at 1676289607431 (+7 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289607431Cleaning up temporary data from old regions at 1676289607489 (+58 ms)Cleaning up detritus from prior splits at 1676289607490 (+1 ms)Running coprocessor post-open hooks at 1676289607498 (+8 ms)Region opened successfully at 1676289607498 2023-02-13 12:00:07,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012., pid=183, masterSystemTime=1676289607349 2023-02-13 12:00:07,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:07,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:07,501 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:07,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:07,501 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289607501"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289607501"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289607501"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289607501"}]},"ts":"1676289607501"} 2023-02-13 12:00:07,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:07,502 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:07,502 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289607502"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289607502"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289607502"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289607502"}]},"ts":"1676289607502"} 2023-02-13 12:00:07,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=180 2023-02-13 12:00:07,509 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=176 2023-02-13 12:00:07,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=180, state=SUCCESS; OpenRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427 in 307 msec 2023-02-13 12:00:07,509 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=176, state=SUCCESS; OpenRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478 in 308 msec 2023-02-13 12:00:07,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, ASSIGN in 472 msec 2023-02-13 12:00:07,511 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=176, resume processing ppid=175 2023-02-13 12:00:07,511 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, ASSIGN in 472 msec 2023-02-13 12:00:07,512 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289607512"}]},"ts":"1676289607512"} 2023-02-13 12:00:07,514 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=ENABLED in hbase:meta 2023-02-13 12:00:07,552 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676289580536 snapshot=name: "snaptb1-1676289580536" table: "testRestoreSnapshot-1676289580536" creation_time: 1676289592259 type: DISABLED version: 2 ttl: 0 ) in 1.0170 sec 2023-02-13 12:00:08,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 12:00:08,044 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676289580536, procId: 175 completed 2023-02-13 12:00:08,045 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(139): Connect 0x39cc9c81 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:00:08,069 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@abe50b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:00:08,072 DEBUG [hconnection-0x6b5c94f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:08,074 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:46402, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:08,078 DEBUG [hconnection-0x6b5c94f5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:08,079 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:41184, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:08,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,307 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:00:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,507 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:00:08,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,756 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:00:08,759 DEBUG [hconnection-0x6b5c94f5-shared-pool-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:08,759 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:37394, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,946 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:00:08,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:08,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,204 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x546668f0 to 127.0.0.1:63172 2023-02-13 12:00:09,204 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:09,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,261 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:00:09,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:10,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,244 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:00:11,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:11,711 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676289580536 2023-02-13 12:00:11,711 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676289580536 2023-02-13 12:00:11,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:11,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 12:00:11,716 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289611716"}]},"ts":"1676289611716"} 2023-02-13 12:00:11,720 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLING in hbase:meta 2023-02-13 12:00:11,735 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676289580536 to state=DISABLING 2023-02-13 12:00:11,736 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN}] 2023-02-13 12:00:11,737 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN 2023-02-13 12:00:11,738 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN 2023-02-13 12:00:11,738 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN 2023-02-13 12:00:11,738 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN 2023-02-13 12:00:11,738 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN 2023-02-13 12:00:11,738 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:11,738 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:11,738 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:11,738 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289611738"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289611738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289611738"}]},"ts":"1676289611738"} 2023-02-13 12:00:11,739 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611738"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289611738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289611738"}]},"ts":"1676289611738"} 2023-02-13 12:00:11,738 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:11,739 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:11,739 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611738"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289611738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289611738"}]},"ts":"1676289611738"} 2023-02-13 12:00:11,738 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611738"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289611738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289611738"}]},"ts":"1676289611738"} 2023-02-13 12:00:11,739 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611739"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289611739"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289611739"}]},"ts":"1676289611739"} 2023-02-13 12:00:11,739 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=194, state=RUNNABLE; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:11,740 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=193, state=RUNNABLE; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:11,741 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=192, state=RUNNABLE; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:11,744 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=191, state=RUNNABLE; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:11,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=190, state=RUNNABLE; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:11,745 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN 2023-02-13 12:00:11,746 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:11,746 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289611746"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289611746"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289611746"}]},"ts":"1676289611746"} 2023-02-13 12:00:11,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=189, state=RUNNABLE; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:11,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:11,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:11,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a40bfc382ab282660f54577d5b7b9575, disabling compactions & flushes 2023-02-13 12:00:11,892 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:11,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:11,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. after waiting 0 ms 2023-02-13 12:00:11,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:11,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing c62ea170afdbf159d444aa319925e012, disabling compactions & flushes 2023-02-13 12:00:11,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:11,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:11,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. after waiting 0 ms 2023-02-13 12:00:11,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:11,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:11,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing e0ee51e8905082f3fc07662881dfd80c, disabling compactions & flushes 2023-02-13 12:00:11,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:11,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:11,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. after waiting 0 ms 2023-02-13 12:00:11,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:11,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:00:11,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:00:11,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575. 2023-02-13 12:00:11,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a40bfc382ab282660f54577d5b7b9575: Waiting for close lock at 1676289611891Running coprocessor pre-close hooks at 1676289611891Disabling compacts and flushes for region at 1676289611891Disabling writes for close at 1676289611893 (+2 ms)Writing region close event to WAL at 1676289611901 (+8 ms)Running coprocessor post-close hooks at 1676289611909 (+8 ms)Closed at 1676289611910 (+1 ms) 2023-02-13 12:00:11,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012. 2023-02-13 12:00:11,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:00:11,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for c62ea170afdbf159d444aa319925e012: Waiting for close lock at 1676289611892Running coprocessor pre-close hooks at 1676289611892Disabling compacts and flushes for region at 1676289611892Disabling writes for close at 1676289611894 (+2 ms)Writing region close event to WAL at 1676289611901 (+7 ms)Running coprocessor post-close hooks at 1676289611910 (+9 ms)Closed at 1676289611913 (+3 ms) 2023-02-13 12:00:11,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c. 2023-02-13 12:00:11,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for e0ee51e8905082f3fc07662881dfd80c: Waiting for close lock at 1676289611900Running coprocessor pre-close hooks at 1676289611900Disabling compacts and flushes for region at 1676289611900Disabling writes for close at 1676289611901 (+1 ms)Writing region close event to WAL at 1676289611907 (+6 ms)Running coprocessor post-close hooks at 1676289611913 (+6 ms)Closed at 1676289611913 2023-02-13 12:00:11,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:11,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:11,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 2b55b9ce69d00ead52d760ba43038e9a, disabling compactions & flushes 2023-02-13 12:00:11,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:11,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:11,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. after waiting 0 ms 2023-02-13 12:00:11,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:11,918 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=a40bfc382ab282660f54577d5b7b9575, regionState=CLOSED 2023-02-13 12:00:11,918 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611917"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289611917"}]},"ts":"1676289611917"} 2023-02-13 12:00:11,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:11,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:11,918 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=c62ea170afdbf159d444aa319925e012, regionState=CLOSED 2023-02-13 12:00:11,918 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611918"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289611918"}]},"ts":"1676289611918"} 2023-02-13 12:00:11,919 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=e0ee51e8905082f3fc07662881dfd80c, regionState=CLOSED 2023-02-13 12:00:11,919 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611919"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289611919"}]},"ts":"1676289611919"} 2023-02-13 12:00:11,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 56eee98ead755df3cb16e7153d171fbf, disabling compactions & flushes 2023-02-13 12:00:11,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:11,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:11,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. after waiting 0 ms 2023-02-13 12:00:11,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:11,925 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:11,926 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:11,930 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=192 2023-02-13 12:00:11,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81f9e5dd49cc8e0565489f1ebf657e0d, disabling compactions & flushes 2023-02-13 12:00:11,936 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=192, state=SUCCESS; CloseRegionProcedure a40bfc382ab282660f54577d5b7b9575, server=jenkins-hbase16.apache.org,34233,1676289577518 in 178 msec 2023-02-13 12:00:11,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:11,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:11,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. after waiting 0 ms 2023-02-13 12:00:11,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:11,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=193 2023-02-13 12:00:11,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=193, state=SUCCESS; CloseRegionProcedure c62ea170afdbf159d444aa319925e012, server=jenkins-hbase16.apache.org,35305,1676289577427 in 180 msec 2023-02-13 12:00:11,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=a40bfc382ab282660f54577d5b7b9575, UNASSIGN in 200 msec 2023-02-13 12:00:11,940 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=191 2023-02-13 12:00:11,940 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=191, state=SUCCESS; CloseRegionProcedure e0ee51e8905082f3fc07662881dfd80c, server=jenkins-hbase16.apache.org,45135,1676289577478 in 188 msec 2023-02-13 12:00:11,942 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=c62ea170afdbf159d444aa319925e012, UNASSIGN in 203 msec 2023-02-13 12:00:11,942 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=e0ee51e8905082f3fc07662881dfd80c, UNASSIGN in 204 msec 2023-02-13 12:00:11,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:00:11,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:00:11,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:00:11,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d. 2023-02-13 12:00:11,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81f9e5dd49cc8e0565489f1ebf657e0d: Waiting for close lock at 1676289611926Running coprocessor pre-close hooks at 1676289611926Disabling compacts and flushes for region at 1676289611926Disabling writes for close at 1676289611937 (+11 ms)Writing region close event to WAL at 1676289611938 (+1 ms)Running coprocessor post-close hooks at 1676289611948 (+10 ms)Closed at 1676289611948 2023-02-13 12:00:11,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a. 2023-02-13 12:00:11,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 2b55b9ce69d00ead52d760ba43038e9a: Waiting for close lock at 1676289611917Running coprocessor pre-close hooks at 1676289611917Disabling compacts and flushes for region at 1676289611917Disabling writes for close at 1676289611918 (+1 ms)Writing region close event to WAL at 1676289611937 (+19 ms)Running coprocessor post-close hooks at 1676289611948 (+11 ms)Closed at 1676289611948 2023-02-13 12:00:11,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:11,950 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=81f9e5dd49cc8e0565489f1ebf657e0d, regionState=CLOSED 2023-02-13 12:00:11,950 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289611950"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289611950"}]},"ts":"1676289611950"} 2023-02-13 12:00:11,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:11,951 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=2b55b9ce69d00ead52d760ba43038e9a, regionState=CLOSED 2023-02-13 12:00:11,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf. 2023-02-13 12:00:11,951 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676289611951"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289611951"}]},"ts":"1676289611951"} 2023-02-13 12:00:11,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 56eee98ead755df3cb16e7153d171fbf: Waiting for close lock at 1676289611918Running coprocessor pre-close hooks at 1676289611918Disabling compacts and flushes for region at 1676289611918Disabling writes for close at 1676289611921 (+3 ms)Writing region close event to WAL at 1676289611938 (+17 ms)Running coprocessor post-close hooks at 1676289611948 (+10 ms)Closed at 1676289611948 2023-02-13 12:00:11,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:11,954 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=56eee98ead755df3cb16e7153d171fbf, regionState=CLOSED 2023-02-13 12:00:11,954 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676289611954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289611954"}]},"ts":"1676289611954"} 2023-02-13 12:00:11,954 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=189 2023-02-13 12:00:11,955 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=189, state=SUCCESS; CloseRegionProcedure 81f9e5dd49cc8e0565489f1ebf657e0d, server=jenkins-hbase16.apache.org,45135,1676289577478 in 205 msec 2023-02-13 12:00:11,955 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=194 2023-02-13 12:00:11,955 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=194, state=SUCCESS; CloseRegionProcedure 2b55b9ce69d00ead52d760ba43038e9a, server=jenkins-hbase16.apache.org,34233,1676289577518 in 214 msec 2023-02-13 12:00:11,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=81f9e5dd49cc8e0565489f1ebf657e0d, UNASSIGN in 218 msec 2023-02-13 12:00:11,956 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=2b55b9ce69d00ead52d760ba43038e9a, UNASSIGN in 219 msec 2023-02-13 12:00:11,957 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=190 2023-02-13 12:00:11,957 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=190, state=SUCCESS; CloseRegionProcedure 56eee98ead755df3cb16e7153d171fbf, server=jenkins-hbase16.apache.org,35305,1676289577427 in 211 msec 2023-02-13 12:00:11,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=190, resume processing ppid=188 2023-02-13 12:00:11,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676289580536, region=56eee98ead755df3cb16e7153d171fbf, UNASSIGN in 221 msec 2023-02-13 12:00:11,958 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289611958"}]},"ts":"1676289611958"} 2023-02-13 12:00:11,960 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676289580536, state=DISABLED in hbase:meta 2023-02-13 12:00:11,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 12:00:11,968 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676289580536 to state=DISABLED 2023-02-13 12:00:11,970 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676289580536 in 257 msec 2023-02-13 12:00:12,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 12:00:12,470 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 188 completed 2023-02-13 12:00:12,472 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$5(2215): Client=jenkins//188.40.62.62 delete testRestoreSnapshot-1676289580536 2023-02-13 12:00:12,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:12,477 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:12,478 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:12,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 12:00:12,486 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:12,486 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:12,486 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:12,486 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:12,486 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:12,486 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:12,488 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits] 2023-02-13 12:00:12,488 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits] 2023-02-13 12:00:12,489 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits] 2023-02-13 12:00:12,489 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits] 2023-02-13 12:00:12,489 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits] 2023-02-13 12:00:12,489 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits] 2023-02-13 12:00:12,492 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606264 currentMtime: 1676289607007 2023-02-13 12:00:12,493 ERROR [HFileArchiver-9] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606264 currentMtime: 1676289606608 2023-02-13 12:00:12,493 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606264 currentMtime: 1676289606621 2023-02-13 12:00:12,493 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c.1676289612488. 2023-02-13 12:00:12,494 INFO [HFileArchiver-9] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23.1676289612489. 2023-02-13 12:00:12,494 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a.1676289612488. 2023-02-13 12:00:12,494 ERROR [HFileArchiver-10] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606264 currentMtime: 1676289606628 2023-02-13 12:00:12,494 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606264 currentMtime: 1676289606608 2023-02-13 12:00:12,494 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-599dabece3764fa8a0da81bc49ad9d1c 2023-02-13 12:00:12,494 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-d847b93f4cdb425190d84b39613c4e23 2023-02-13 12:00:12,494 INFO [HFileArchiver-10] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0.1676289612489. 2023-02-13 12:00:12,495 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-29f3eaf3b2c946079123a6bd68c2201a 2023-02-13 12:00:12,495 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606265 currentMtime: 1676289606608 2023-02-13 12:00:12,495 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3.1676289612489. 2023-02-13 12:00:12,495 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-1e304a838f584997acee189c66a2bea0 2023-02-13 12:00:12,495 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480.1676289612489. 2023-02-13 12:00:12,496 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606266 currentMtime: 1676289607010 2023-02-13 12:00:12,496 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-6f4565b390274a5e9c444537d8b9bca3 2023-02-13 12:00:12,496 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-276801e0d81b49fd9821e25bfaecd480 2023-02-13 12:00:12,496 ERROR [HFileArchiver-9] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606267 currentMtime: 1676289606623 2023-02-13 12:00:12,496 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627.1676289612488. 2023-02-13 12:00:12,496 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606267 currentMtime: 1676289606629 2023-02-13 12:00:12,497 INFO [HFileArchiver-9] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca.1676289612489. 2023-02-13 12:00:12,497 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf.1676289612488. 2023-02-13 12:00:12,497 ERROR [HFileArchiver-10] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606266 currentMtime: 1676289606631 2023-02-13 12:00:12,497 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/cf/testRestoreSnapshot-1676289580536=e0ee51e8905082f3fc07662881dfd80c-c193a7455d3d48f29f60b9563d1f1627 2023-02-13 12:00:12,498 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606266 currentMtime: 1676289606628 2023-02-13 12:00:12,498 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/cf/testRestoreSnapshot-1676289580536=2b55b9ce69d00ead52d760ba43038e9a-ff22144d81504ead9754a1bdee26e1cf 2023-02-13 12:00:12,499 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/cf/testRestoreSnapshot-1676289580536=81f9e5dd49cc8e0565489f1ebf657e0d-fe54fb5b90734f3bb8a4e7c25ecba3ca 2023-02-13 12:00:12,499 INFO [HFileArchiver-10] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105.1676289612489. 2023-02-13 12:00:12,499 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606266 currentMtime: 1676289606625 2023-02-13 12:00:12,499 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4.1676289612489. 2023-02-13 12:00:12,500 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b.1676289612489. 2023-02-13 12:00:12,504 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/cf/testRestoreSnapshot-1676289580536=56eee98ead755df3cb16e7153d171fbf-efd9c8e078ce4850b8dcbe3b73c8a105 2023-02-13 12:00:12,504 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/cf/testRestoreSnapshot-1676289580536=c62ea170afdbf159d444aa319925e012-a5cf918d60df457f8c1a52e0029a35e4 2023-02-13 12:00:12,504 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/cf/testRestoreSnapshot-1676289580536=a40bfc382ab282660f54577d5b7b9575-d71da190e77346458dd9bdfc6068675b 2023-02-13 12:00:12,505 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c/recovered.edits/18.seqid 2023-02-13 12:00:12,505 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a/recovered.edits/18.seqid 2023-02-13 12:00:12,505 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d/recovered.edits/18.seqid 2023-02-13 12:00:12,506 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/e0ee51e8905082f3fc07662881dfd80c 2023-02-13 12:00:12,506 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/2b55b9ce69d00ead52d760ba43038e9a 2023-02-13 12:00:12,506 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/81f9e5dd49cc8e0565489f1ebf657e0d 2023-02-13 12:00:12,506 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf/recovered.edits/18.seqid 2023-02-13 12:00:12,507 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012/recovered.edits/18.seqid 2023-02-13 12:00:12,507 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575/recovered.edits/18.seqid 2023-02-13 12:00:12,507 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/56eee98ead755df3cb16e7153d171fbf 2023-02-13 12:00:12,507 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/c62ea170afdbf159d444aa319925e012 2023-02-13 12:00:12,507 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testRestoreSnapshot-1676289580536/a40bfc382ab282660f54577d5b7b9575 2023-02-13 12:00:12,507 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676289580536 regions 2023-02-13 12:00:12,508 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be 2023-02-13 12:00:12,508 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf] 2023-02-13 12:00:12,510 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606275 currentMtime: 1676289606556 2023-02-13 12:00:12,510 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136.1676289612508. 2023-02-13 12:00:12,511 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c202302138bf50d7a07844eebbfa70e24328d4136 2023-02-13 12:00:12,511 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606277 currentMtime: 1676289606558 2023-02-13 12:00:12,512 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74.1676289612508. 2023-02-13 12:00:12,513 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-a87ff679a2f3e71d9181a67b7542122c20230213b7204d420f8340c5bc2dde7e8cddfa74 2023-02-13 12:00:12,514 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606278 currentMtime: 1676289606560 2023-02-13 12:00:12,514 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea.1676289612508. 2023-02-13 12:00:12,514 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b20230213114b80286b274bb9ab57c4842ceadfea 2023-02-13 12:00:12,515 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606279 currentMtime: 1676289606562 2023-02-13 12:00:12,516 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8.1676289612508. 2023-02-13 12:00:12,516 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c4ca4238a0b923820dcc509a6f75849b2023021322734db13ed74aedb9ef51aa0ee85ac8 2023-02-13 12:00:12,517 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606279 currentMtime: 1676289606563 2023-02-13 12:00:12,517 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1.1676289612508. 2023-02-13 12:00:12,518 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c202302138218d6e6396543cbaa26535d5f8b74b1 2023-02-13 12:00:12,519 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606280 currentMtime: 1676289606565 2023-02-13 12:00:12,519 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c.1676289612508. 2023-02-13 12:00:12,519 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-c81e728d9d4c2f636f067f89cc14862c20230213b0bb7e4a772246c1ada70fec288ac62c 2023-02-13 12:00:12,521 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606281 currentMtime: 1676289606566 2023-02-13 12:00:12,521 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d.1676289612508. 2023-02-13 12:00:12,522 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e2023021368dd141eb15445d09ee24fbeab87d88d 2023-02-13 12:00:12,523 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606282 currentMtime: 1676289606567 2023-02-13 12:00:12,523 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96.1676289612508. 2023-02-13 12:00:12,524 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-d41d8cd98f00b204e9800998ecf8427e20230213702844471ea644b187452392b450ac96 2023-02-13 12:00:12,525 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606282 currentMtime: 1676289606569 2023-02-13 12:00:12,525 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2.1676289612508. 2023-02-13 12:00:12,526 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302131ea62ce7954f4ce48f5bf68ae32789d2 2023-02-13 12:00:12,527 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606283 currentMtime: 1676289606570 2023-02-13 12:00:12,527 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7.1676289612508. 2023-02-13 12:00:12,528 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-e4da3b7fbbce2345d7772b0674a318d5202302137ab3bedb2e9e495481077aae857cb2a7 2023-02-13 12:00:12,529 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606284 currentMtime: 1676289606572 2023-02-13 12:00:12,530 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352.1676289612508. 2023-02-13 12:00:12,530 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213d7eda89e7572417bb3b666ace4829352 2023-02-13 12:00:12,531 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef. archiveLen: 0 currentLen: 0 archiveMtime: 1676289606285 currentMtime: 1676289606573 2023-02-13 12:00:12,532 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef.1676289612508. 2023-02-13 12:00:12,533 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be/cf/testRestoreSnapshot-1676289580536=aab21f680c1fff98d3160c527e41e1be-eccbc87e4b5ce2fe28308fd9f2a7baf320230213ff29d38a91754ea79cb04b89476a46ef 2023-02-13 12:00:12,533 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testRestoreSnapshot-1676289580536/aab21f680c1fff98d3160c527e41e1be 2023-02-13 12:00:12,536 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:12,539 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676289580536 from hbase:meta 2023-02-13 12:00:12,541 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676289580536' descriptor. 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676289580536' from region states. 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289612543"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289612543"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289612543"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289612543"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289612543"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:12,543 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289612543"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:12,548 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:00:12,549 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 81f9e5dd49cc8e0565489f1ebf657e0d, NAME => 'testRestoreSnapshot-1676289580536,,1676289580570.81f9e5dd49cc8e0565489f1ebf657e0d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 56eee98ead755df3cb16e7153d171fbf, NAME => 'testRestoreSnapshot-1676289580536,1,1676289580570.56eee98ead755df3cb16e7153d171fbf.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => e0ee51e8905082f3fc07662881dfd80c, NAME => 'testRestoreSnapshot-1676289580536,2,1676289580570.e0ee51e8905082f3fc07662881dfd80c.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => a40bfc382ab282660f54577d5b7b9575, NAME => 'testRestoreSnapshot-1676289580536,3,1676289580570.a40bfc382ab282660f54577d5b7b9575.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => c62ea170afdbf159d444aa319925e012, NAME => 'testRestoreSnapshot-1676289580536,4,1676289580570.c62ea170afdbf159d444aa319925e012.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 2b55b9ce69d00ead52d760ba43038e9a, NAME => 'testRestoreSnapshot-1676289580536,5,1676289580570.2b55b9ce69d00ead52d760ba43038e9a.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 12:00:12,549 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676289580536' as deleted. 2023-02-13 12:00:12,549 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676289580536","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676289612549"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:12,551 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676289580536 state from META 2023-02-13 12:00:12,561 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 2023-02-13 12:00:12,563 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676289580536 in 88 msec 2023-02-13 12:00:12,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 12:00:12,732 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676289580536, procId: 201 completed 2023-02-13 12:00:12,749 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "emptySnaptb-1676289580536" 2023-02-13 12:00:12,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676289580536 2023-02-13 12:00:12,754 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "snaptb0-1676289580536" 2023-02-13 12:00:12,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676289580536 2023-02-13 12:00:12,758 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "snaptb1-1676289580536" 2023-02-13 12:00:12,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676289580536 2023-02-13 12:00:12,794 INFO [Listener at localhost.localdomain/36409] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=409 (was 363) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63172@0x39cc9c81-SendThread(127.0.0.1:63172) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:59838 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6b5c94f5-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63172@0x39cc9c81-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:49542 [Waiting for operation #496] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6136c840-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-541515498_15 at /127.0.0.1:58490 [Waiting for operation #513] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63172@0x39cc9c81 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/623046442.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:58474 [Waiting for operation #535] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:52782 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:49556 [Waiting for operation #489] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=793 (was 697) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=446 (was 280) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=4515 (was 6109) 2023-02-13 12:00:12,808 INFO [Listener at localhost.localdomain/36409] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=409, OpenFileDescriptor=793, MaxFileDescriptor=60000, SystemLoadAverage=446, ProcessCount=171, AvailableMemoryMB=4515 2023-02-13 12:00:12,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$4(2102): Client=jenkins//188.40.62.62 create 'testCorruptedSnapshot-1676289612809', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 12:00:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:12,814 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 12:00:12,814 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676289612809" procId is: 202 2023-02-13 12:00:12,814 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 12:00:12,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 12:00:12,819 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:12,819 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:12,819 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:12,819 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:12,819 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:12,819 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 empty. 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca empty. 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 empty. 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 empty. 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a empty. 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:12,820 DEBUG [HFileArchiver-9] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb empty. 2023-02-13 12:00:12,821 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:12,821 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:12,821 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:12,821 DEBUG [HFileArchiver-9] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:12,821 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676289612809 regions 2023-02-13 12:00:12,834 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/.tabledesc/.tableinfo.0000000001 2023-02-13 12:00:12,836 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(7671): creating {ENCODED => 9bdb004dd508672ca8f98b4ad3d79f3a, NAME => 'testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676289612809', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:12,836 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(7671): creating {ENCODED => f5abf54abb980cb8f138bde5b646e3c0, NAME => 'testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676289612809', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:12,836 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(7671): creating {ENCODED => 48d6107ef6392fc0224ecd69dad7d2f3, NAME => 'testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676289612809', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:12,850 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:12,850 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1603): Closing 48d6107ef6392fc0224ecd69dad7d2f3, disabling compactions & flushes 2023-02-13 12:00:12,850 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:12,851 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:12,851 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. after waiting 0 ms 2023-02-13 12:00:12,851 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:12,851 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:12,851 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1557): Region close journal for 48d6107ef6392fc0224ecd69dad7d2f3: Waiting for close lock at 1676289612850Disabling compacts and flushes for region at 1676289612850Disabling writes for close at 1676289612851 (+1 ms)Writing region close event to WAL at 1676289612851Closed at 1676289612851 2023-02-13 12:00:12,851 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(7671): creating {ENCODED => a0478ecfb8560aee2a65d010b27747ca, NAME => 'testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676289612809', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1603): Closing 9bdb004dd508672ca8f98b4ad3d79f3a, disabling compactions & flushes 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:12,853 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1603): Closing f5abf54abb980cb8f138bde5b646e3c0, disabling compactions & flushes 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:12,853 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. after waiting 0 ms 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. after waiting 0 ms 2023-02-13 12:00:12,853 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1557): Region close journal for 9bdb004dd508672ca8f98b4ad3d79f3a: Waiting for close lock at 1676289612853Disabling compacts and flushes for region at 1676289612853Disabling writes for close at 1676289612853Writing region close event to WAL at 1676289612853Closed at 1676289612853 2023-02-13 12:00:12,853 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:12,853 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1557): Region close journal for f5abf54abb980cb8f138bde5b646e3c0: Waiting for close lock at 1676289612853Disabling compacts and flushes for region at 1676289612853Disabling writes for close at 1676289612853Writing region close event to WAL at 1676289612853Closed at 1676289612853 2023-02-13 12:00:12,854 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(7671): creating {ENCODED => 643a6d6a4f0efd58319e8e38b8b918fb, NAME => 'testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676289612809', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:12,854 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(7671): creating {ENCODED => 81a6a2c6c5a450fab5269836430582d6, NAME => 'testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676289612809', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp 2023-02-13 12:00:12,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:12,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1603): Closing a0478ecfb8560aee2a65d010b27747ca, disabling compactions & flushes 2023-02-13 12:00:12,862 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:12,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:12,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. after waiting 0 ms 2023-02-13 12:00:12,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:12,862 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:12,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-2] regionserver.HRegion(1557): Region close journal for a0478ecfb8560aee2a65d010b27747ca: Waiting for close lock at 1676289612862Disabling compacts and flushes for region at 1676289612862Disabling writes for close at 1676289612862Writing region close event to WAL at 1676289612862Closed at 1676289612862 2023-02-13 12:00:12,864 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:12,864 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:12,864 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1603): Closing 643a6d6a4f0efd58319e8e38b8b918fb, disabling compactions & flushes 2023-02-13 12:00:12,864 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1603): Closing 81a6a2c6c5a450fab5269836430582d6, disabling compactions & flushes 2023-02-13 12:00:12,864 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:12,864 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:12,864 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:12,865 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:12,865 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. after waiting 0 ms 2023-02-13 12:00:12,865 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. after waiting 0 ms 2023-02-13 12:00:12,865 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:12,865 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:12,865 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:12,865 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:12,865 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-1] regionserver.HRegion(1557): Region close journal for 643a6d6a4f0efd58319e8e38b8b918fb: Waiting for close lock at 1676289612864Disabling compacts and flushes for region at 1676289612864Disabling writes for close at 1676289612865 (+1 ms)Writing region close event to WAL at 1676289612865Closed at 1676289612865 2023-02-13 12:00:12,865 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676289612809-pool-0] regionserver.HRegion(1557): Region close journal for 81a6a2c6c5a450fab5269836430582d6: Waiting for close lock at 1676289612864Disabling compacts and flushes for region at 1676289612864Disabling writes for close at 1676289612865 (+1 ms)Writing region close event to WAL at 1676289612865Closed at 1676289612865 2023-02-13 12:00:12,867 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 12:00:12,867 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289612867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289612867"}]},"ts":"1676289612867"} 2023-02-13 12:00:12,867 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289612867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289612867"}]},"ts":"1676289612867"} 2023-02-13 12:00:12,868 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289612867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289612867"}]},"ts":"1676289612867"} 2023-02-13 12:00:12,868 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289612867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289612867"}]},"ts":"1676289612867"} 2023-02-13 12:00:12,868 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289612867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289612867"}]},"ts":"1676289612867"} 2023-02-13 12:00:12,868 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289612867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289612867"}]},"ts":"1676289612867"} 2023-02-13 12:00:12,869 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:00:12,870 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 12:00:12,870 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289612870"}]},"ts":"1676289612870"} 2023-02-13 12:00:12,871 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=ENABLING in hbase:meta 2023-02-13 12:00:12,891 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:00:12,894 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:00:12,894 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:00:12,894 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:00:12,894 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:00:12,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN}] 2023-02-13 12:00:12,896 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN 2023-02-13 12:00:12,896 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN 2023-02-13 12:00:12,896 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN 2023-02-13 12:00:12,896 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN 2023-02-13 12:00:12,897 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN 2023-02-13 12:00:12,897 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=false 2023-02-13 12:00:12,897 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=false 2023-02-13 12:00:12,897 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=false 2023-02-13 12:00:12,897 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=false 2023-02-13 12:00:12,898 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=false 2023-02-13 12:00:12,898 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN 2023-02-13 12:00:12,898 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=false 2023-02-13 12:00:13,047 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:00:13,054 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:13,054 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:13,054 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613054"}]},"ts":"1676289613054"} 2023-02-13 12:00:13,054 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613054"}]},"ts":"1676289613054"} 2023-02-13 12:00:13,054 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:13,054 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:13,055 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613054"}]},"ts":"1676289613054"} 2023-02-13 12:00:13,055 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613054"}]},"ts":"1676289613054"} 2023-02-13 12:00:13,054 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:13,055 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613054"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613054"}]},"ts":"1676289613054"} 2023-02-13 12:00:13,056 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=203, state=RUNNABLE; OpenRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:13,057 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=206, state=RUNNABLE; OpenRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:13,058 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=207, state=RUNNABLE; OpenRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:13,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=208, state=RUNNABLE; OpenRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:13,059 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=204, state=RUNNABLE; OpenRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:13,059 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:13,060 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613059"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613059"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613059"}]},"ts":"1676289613059"} 2023-02-13 12:00:13,061 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=205, state=RUNNABLE; OpenRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:13,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 12:00:13,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:13,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:13,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a6a2c6c5a450fab5269836430582d6, NAME => 'testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:00:13,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 48d6107ef6392fc0224ecd69dad7d2f3, NAME => 'testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:00:13,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:13,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bdb004dd508672ca8f98b4ad3d79f3a, NAME => 'testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:13,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,217 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,217 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,217 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,218 DEBUG [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf 2023-02-13 12:00:13,218 DEBUG [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf 2023-02-13 12:00:13,218 DEBUG [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf 2023-02-13 12:00:13,218 DEBUG [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf 2023-02-13 12:00:13,218 DEBUG [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf 2023-02-13 12:00:13,218 DEBUG [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf 2023-02-13 12:00:13,219 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 48d6107ef6392fc0224ecd69dad7d2f3 columnFamilyName cf 2023-02-13 12:00:13,219 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bdb004dd508672ca8f98b4ad3d79f3a columnFamilyName cf 2023-02-13 12:00:13,219 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a6a2c6c5a450fab5269836430582d6 columnFamilyName cf 2023-02-13 12:00:13,220 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] regionserver.HStore(310): Store=48d6107ef6392fc0224ecd69dad7d2f3/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:13,220 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] regionserver.HStore(310): Store=9bdb004dd508672ca8f98b4ad3d79f3a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:13,220 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] regionserver.HStore(310): Store=81a6a2c6c5a450fab5269836430582d6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:13,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:00:13,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:00:13,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:00:13,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 48d6107ef6392fc0224ecd69dad7d2f3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59052811, jitterRate=-0.12004454433917999}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:13,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 9bdb004dd508672ca8f98b4ad3d79f3a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70021052, jitterRate=0.04339498281478882}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:13,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 48d6107ef6392fc0224ecd69dad7d2f3: Running coprocessor pre-open hook at 1676289613215Writing region info on filesystem at 1676289613215Initializing all the Stores at 1676289613216 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289613217 (+1 ms)Cleaning up temporary data from old regions at 1676289613221 (+4 ms)Cleaning up detritus from prior splits at 1676289613221Running coprocessor post-open hooks at 1676289613225 (+4 ms)Region opened successfully at 1676289613225 2023-02-13 12:00:13,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a6a2c6c5a450fab5269836430582d6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62496665, jitterRate=-0.06872712075710297}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:13,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 9bdb004dd508672ca8f98b4ad3d79f3a: Running coprocessor pre-open hook at 1676289613216Writing region info on filesystem at 1676289613216Initializing all the Stores at 1676289613217 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289613217Cleaning up temporary data from old regions at 1676289613221 (+4 ms)Cleaning up detritus from prior splits at 1676289613222 (+1 ms)Running coprocessor post-open hooks at 1676289613225 (+3 ms)Region opened successfully at 1676289613225 2023-02-13 12:00:13,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a6a2c6c5a450fab5269836430582d6: Running coprocessor pre-open hook at 1676289613216Writing region info on filesystem at 1676289613216Initializing all the Stores at 1676289613217 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289613217Cleaning up temporary data from old regions at 1676289613221 (+4 ms)Cleaning up detritus from prior splits at 1676289613222 (+1 ms)Running coprocessor post-open hooks at 1676289613225 (+3 ms)Region opened successfully at 1676289613225 2023-02-13 12:00:13,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3., pid=214, masterSystemTime=1676289613208 2023-02-13 12:00:13,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a., pid=213, masterSystemTime=1676289613209 2023-02-13 12:00:13,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6., pid=212, masterSystemTime=1676289613208 2023-02-13 12:00:13,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:13,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:13,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:13,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a0478ecfb8560aee2a65d010b27747ca, NAME => 'testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:00:13,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:13,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,228 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:13,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:13,229 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613228"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289613228"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289613228"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289613228"}]},"ts":"1676289613228"} 2023-02-13 12:00:13,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:13,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:13,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 643a6d6a4f0efd58319e8e38b8b918fb, NAME => 'testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:00:13,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,229 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:13,229 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,229 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613229"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289613229"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289613229"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289613229"}]},"ts":"1676289613229"} 2023-02-13 12:00:13,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:13,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:13,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:13,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f5abf54abb980cb8f138bde5b646e3c0, NAME => 'testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:00:13,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:13,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,231 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:13,231 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613230"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289613230"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289613230"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289613230"}]},"ts":"1676289613230"} 2023-02-13 12:00:13,231 DEBUG [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf 2023-02-13 12:00:13,231 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,231 DEBUG [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf 2023-02-13 12:00:13,231 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,232 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a0478ecfb8560aee2a65d010b27747ca columnFamilyName cf 2023-02-13 12:00:13,233 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] regionserver.HStore(310): Store=a0478ecfb8560aee2a65d010b27747ca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:13,234 DEBUG [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf 2023-02-13 12:00:13,234 DEBUG [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf 2023-02-13 12:00:13,234 DEBUG [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf 2023-02-13 12:00:13,234 DEBUG [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf 2023-02-13 12:00:13,234 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f5abf54abb980cb8f138bde5b646e3c0 columnFamilyName cf 2023-02-13 12:00:13,234 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 643a6d6a4f0efd58319e8e38b8b918fb columnFamilyName cf 2023-02-13 12:00:13,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,236 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=205 2023-02-13 12:00:13,237 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=205, state=SUCCESS; OpenRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518 in 170 msec 2023-02-13 12:00:13,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=204 2023-02-13 12:00:13,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=204, state=SUCCESS; OpenRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 173 msec 2023-02-13 12:00:13,238 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN in 343 msec 2023-02-13 12:00:13,238 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] regionserver.HStore(310): Store=643a6d6a4f0efd58319e8e38b8b918fb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:13,238 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN in 343 msec 2023-02-13 12:00:13,239 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] regionserver.HStore(310): Store=f5abf54abb980cb8f138bde5b646e3c0/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:13,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:00:13,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 643a6d6a4f0efd58319e8e38b8b918fb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63685678, jitterRate=-0.05100944638252258}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:13,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 643a6d6a4f0efd58319e8e38b8b918fb: Running coprocessor pre-open hook at 1676289613230Writing region info on filesystem at 1676289613230Initializing all the Stores at 1676289613231 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289613231Cleaning up temporary data from old regions at 1676289613242 (+11 ms)Cleaning up detritus from prior splits at 1676289613242Running coprocessor post-open hooks at 1676289613245 (+3 ms)Region opened successfully at 1676289613245 2023-02-13 12:00:13,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb., pid=211, masterSystemTime=1676289613209 2023-02-13 12:00:13,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:00:13,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a0478ecfb8560aee2a65d010b27747ca; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62271505, jitterRate=-0.0720822662115097}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:13,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a0478ecfb8560aee2a65d010b27747ca: Running coprocessor pre-open hook at 1676289613228Writing region info on filesystem at 1676289613228Initializing all the Stores at 1676289613229 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289613229Cleaning up temporary data from old regions at 1676289613238 (+9 ms)Cleaning up detritus from prior splits at 1676289613248 (+10 ms)Running coprocessor post-open hooks at 1676289613252 (+4 ms)Region opened successfully at 1676289613252 2023-02-13 12:00:13,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:00:13,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened f5abf54abb980cb8f138bde5b646e3c0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71624869, jitterRate=0.06729371845722198}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:13,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for f5abf54abb980cb8f138bde5b646e3c0: Running coprocessor pre-open hook at 1676289613230Writing region info on filesystem at 1676289613230Initializing all the Stores at 1676289613231 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289613231Cleaning up temporary data from old regions at 1676289613249 (+18 ms)Cleaning up detritus from prior splits at 1676289613249Running coprocessor post-open hooks at 1676289613253 (+4 ms)Region opened successfully at 1676289613253 2023-02-13 12:00:13,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0., pid=209, masterSystemTime=1676289613208 2023-02-13 12:00:13,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:13,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:13,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca., pid=210, masterSystemTime=1676289613208 2023-02-13 12:00:13,260 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:13,260 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613260"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289613260"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289613260"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289613260"}]},"ts":"1676289613260"} 2023-02-13 12:00:13,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=208 2023-02-13 12:00:13,261 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:13,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=208, state=SUCCESS; OpenRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478 in 186 msec 2023-02-13 12:00:13,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,261 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613261"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289613261"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289613261"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289613261"}]},"ts":"1676289613261"} 2023-02-13 12:00:13,264 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN in 367 msec 2023-02-13 12:00:13,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:13,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:13,265 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:13,265 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613265"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289613265"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289613265"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289613265"}]},"ts":"1676289613265"} 2023-02-13 12:00:13,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=207 2023-02-13 12:00:13,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=207, state=SUCCESS; OpenRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427 in 204 msec 2023-02-13 12:00:13,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=203 2023-02-13 12:00:13,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=203, state=SUCCESS; OpenRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478 in 208 msec 2023-02-13 12:00:13,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN in 372 msec 2023-02-13 12:00:13,267 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN in 372 msec 2023-02-13 12:00:13,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=206 2023-02-13 12:00:13,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=206, state=SUCCESS; OpenRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518 in 210 msec 2023-02-13 12:00:13,270 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=206, resume processing ppid=202 2023-02-13 12:00:13,270 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN in 375 msec 2023-02-13 12:00:13,271 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 12:00:13,271 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289613271"}]},"ts":"1676289613271"} 2023-02-13 12:00:13,273 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=ENABLED in hbase:meta 2023-02-13 12:00:13,344 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 12:00:13,346 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676289612809 in 533 msec 2023-02-13 12:00:13,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 12:00:13,568 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676289612809, procId: 202 completed 2023-02-13 12:00:13,570 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676289612809 2023-02-13 12:00:13,570 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,574 INFO [Listener at localhost.localdomain/36409] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:00:13,590 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676289612809 2023-02-13 12:00:13,590 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testCorruptedSnapshot-1676289612809 2023-02-13 12:00:13,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:13,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 12:00:13,593 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289613593"}]},"ts":"1676289613593"} 2023-02-13 12:00:13,594 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=DISABLING in hbase:meta 2023-02-13 12:00:13,602 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676289612809 to state=DISABLING 2023-02-13 12:00:13,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN}] 2023-02-13 12:00:13,604 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN 2023-02-13 12:00:13,604 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN 2023-02-13 12:00:13,604 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN 2023-02-13 12:00:13,604 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN 2023-02-13 12:00:13,604 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN 2023-02-13 12:00:13,605 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:13,605 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:13,605 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613605"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613605"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613605"}]},"ts":"1676289613605"} 2023-02-13 12:00:13,605 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613605"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613605"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613605"}]},"ts":"1676289613605"} 2023-02-13 12:00:13,605 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:13,605 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:13,605 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:13,605 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613605"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613605"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613605"}]},"ts":"1676289613605"} 2023-02-13 12:00:13,605 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613605"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613605"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613605"}]},"ts":"1676289613605"} 2023-02-13 12:00:13,605 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613605"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613605"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613605"}]},"ts":"1676289613605"} 2023-02-13 12:00:13,607 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=218, state=RUNNABLE; CloseRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:13,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=217, state=RUNNABLE; CloseRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:13,608 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=220, state=RUNNABLE; CloseRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:13,609 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=221, state=RUNNABLE; CloseRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:13,610 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN 2023-02-13 12:00:13,611 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=219, state=RUNNABLE; CloseRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:13,612 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:13,612 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613612"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289613612"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289613612"}]},"ts":"1676289613612"} 2023-02-13 12:00:13,613 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:13,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 48d6107ef6392fc0224ecd69dad7d2f3, disabling compactions & flushes 2023-02-13 12:00:13,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:13,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:13,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. after waiting 0 ms 2023-02-13 12:00:13,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:13,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 643a6d6a4f0efd58319e8e38b8b918fb, disabling compactions & flushes 2023-02-13 12:00:13,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:13,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:13,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. after waiting 0 ms 2023-02-13 12:00:13,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:13,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a6a2c6c5a450fab5269836430582d6, disabling compactions & flushes 2023-02-13 12:00:13,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:13,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:13,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. after waiting 0 ms 2023-02-13 12:00:13,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:13,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:00:13,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:00:13,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:13,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 48d6107ef6392fc0224ecd69dad7d2f3: Waiting for close lock at 1676289613759Running coprocessor pre-close hooks at 1676289613759Disabling compacts and flushes for region at 1676289613759Disabling writes for close at 1676289613760 (+1 ms)Writing region close event to WAL at 1676289613760Running coprocessor post-close hooks at 1676289613765 (+5 ms)Closed at 1676289613765 2023-02-13 12:00:13,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:13,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 643a6d6a4f0efd58319e8e38b8b918fb: Waiting for close lock at 1676289613760Running coprocessor pre-close hooks at 1676289613760Disabling compacts and flushes for region at 1676289613760Disabling writes for close at 1676289613760Writing region close event to WAL at 1676289613761 (+1 ms)Running coprocessor post-close hooks at 1676289613766 (+5 ms)Closed at 1676289613766 2023-02-13 12:00:13,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:13,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a0478ecfb8560aee2a65d010b27747ca, disabling compactions & flushes 2023-02-13 12:00:13,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:13,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:13,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. after waiting 0 ms 2023-02-13 12:00:13,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:13,768 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=CLOSED 2023-02-13 12:00:13,768 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613768"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289613768"}]},"ts":"1676289613768"} 2023-02-13 12:00:13,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:13,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 9bdb004dd508672ca8f98b4ad3d79f3a, disabling compactions & flushes 2023-02-13 12:00:13,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:13,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:13,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. after waiting 0 ms 2023-02-13 12:00:13,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:13,769 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=CLOSED 2023-02-13 12:00:13,769 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289613769"}]},"ts":"1676289613769"} 2023-02-13 12:00:13,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:00:13,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:13,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a6a2c6c5a450fab5269836430582d6: Waiting for close lock at 1676289613764Running coprocessor pre-close hooks at 1676289613764Disabling compacts and flushes for region at 1676289613764Disabling writes for close at 1676289613764Writing region close event to WAL at 1676289613764Running coprocessor post-close hooks at 1676289613769 (+5 ms)Closed at 1676289613769 2023-02-13 12:00:13,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:13,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing f5abf54abb980cb8f138bde5b646e3c0, disabling compactions & flushes 2023-02-13 12:00:13,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. after waiting 0 ms 2023-02-13 12:00:13,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,772 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=220 2023-02-13 12:00:13,772 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=CLOSED 2023-02-13 12:00:13,772 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=220, state=SUCCESS; CloseRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427 in 162 msec 2023-02-13 12:00:13,772 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=218 2023-02-13 12:00:13,772 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613772"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289613772"}]},"ts":"1676289613772"} 2023-02-13 12:00:13,772 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=218, state=SUCCESS; CloseRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518 in 162 msec 2023-02-13 12:00:13,773 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN in 170 msec 2023-02-13 12:00:13,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN in 170 msec 2023-02-13 12:00:13,774 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=221 2023-02-13 12:00:13,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=221, state=SUCCESS; CloseRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478 in 164 msec 2023-02-13 12:00:13,775 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN in 172 msec 2023-02-13 12:00:13,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:00:13,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:00:13,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:00:13,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:13,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a0478ecfb8560aee2a65d010b27747ca: Waiting for close lock at 1676289613767Running coprocessor pre-close hooks at 1676289613767Disabling compacts and flushes for region at 1676289613767Disabling writes for close at 1676289613768 (+1 ms)Writing region close event to WAL at 1676289613769 (+1 ms)Running coprocessor post-close hooks at 1676289613776 (+7 ms)Closed at 1676289613776 2023-02-13 12:00:13,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:13,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:13,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 9bdb004dd508672ca8f98b4ad3d79f3a: Waiting for close lock at 1676289613768Running coprocessor pre-close hooks at 1676289613768Disabling compacts and flushes for region at 1676289613768Disabling writes for close at 1676289613768Writing region close event to WAL at 1676289613769 (+1 ms)Running coprocessor post-close hooks at 1676289613776 (+7 ms)Closed at 1676289613776 2023-02-13 12:00:13,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for f5abf54abb980cb8f138bde5b646e3c0: Waiting for close lock at 1676289613771Running coprocessor pre-close hooks at 1676289613771Disabling compacts and flushes for region at 1676289613771Disabling writes for close at 1676289613771Writing region close event to WAL at 1676289613772 (+1 ms)Running coprocessor post-close hooks at 1676289613776 (+4 ms)Closed at 1676289613776 2023-02-13 12:00:13,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:13,778 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=CLOSED 2023-02-13 12:00:13,778 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613778"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289613778"}]},"ts":"1676289613778"} 2023-02-13 12:00:13,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:13,779 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=CLOSED 2023-02-13 12:00:13,779 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289613779"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289613779"}]},"ts":"1676289613779"} 2023-02-13 12:00:13,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:13,779 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=CLOSED 2023-02-13 12:00:13,779 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289613779"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289613779"}]},"ts":"1676289613779"} 2023-02-13 12:00:13,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=219 2023-02-13 12:00:13,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=219, state=SUCCESS; CloseRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518 in 169 msec 2023-02-13 12:00:13,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=217 2023-02-13 12:00:13,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=217, state=SUCCESS; CloseRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 173 msec 2023-02-13 12:00:13,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN in 179 msec 2023-02-13 12:00:13,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN in 179 msec 2023-02-13 12:00:13,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-13 12:00:13,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478 in 168 msec 2023-02-13 12:00:13,783 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=216, resume processing ppid=215 2023-02-13 12:00:13,783 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN in 180 msec 2023-02-13 12:00:13,784 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289613784"}]},"ts":"1676289613784"} 2023-02-13 12:00:13,785 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=DISABLED in hbase:meta 2023-02-13 12:00:13,793 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676289612809 to state=DISABLED 2023-02-13 12:00:13,795 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676289612809 in 204 msec 2023-02-13 12:00:13,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 12:00:13,845 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676289612809, procId: 215 completed 2023-02-13 12:00:13,846 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } 2023-02-13 12:00:13,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289613846 (current time:1676289613846). 2023-02-13 12:00:13,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:00:13,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x56c92bac to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:00:13,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18a5d879, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:00:13,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:13,866 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52160, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:13,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c92bac to 127.0.0.1:63172 2023-02-13 12:00:13,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:13,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:00:13,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } 2023-02-13 12:00:13,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE 2023-02-13 12:00:13,872 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE 2023-02-13 12:00:13,873 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE; timeout=600000, timestamp=1676290213873 2023-02-13 12:00:13,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } 2023-02-13 12:00:13,873 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676289612809 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676289612809 2023-02-13 12:00:13,873 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:00:13,873 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:00:13,884 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=DISABLED ttl=0 } 2023-02-13 12:00:13,888 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:13,888 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:13,889 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:13,889 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:13,889 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:13,889 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:13,892 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:13,896 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:13,896 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:13,896 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:13,897 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:13,897 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:13,897 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:13,897 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:00:13,932 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39cc9c81 to 127.0.0.1:63172 2023-02-13 12:00:13,932 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:14,123 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:00:14,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } is done 2023-02-13 12:00:14,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 12:00:14,125 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 12:00:14,349 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=DISABLED ttl=0 } as finished. 2023-02-13 12:00:14,349 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676289612809 2023-02-13 12:00:14,349 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289612809 2023-02-13 12:00:14,372 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289612809 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/emptySnaptb-1676289612809 2023-02-13 12:00:14,389 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676289612809 of table testCorruptedSnapshot-1676289612809 completed 2023-02-13 12:00:14,389 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289612809 2023-02-13 12:00:14,390 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/emptySnaptb-1676289612809 2023-02-13 12:00:14,390 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676289612809 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676289612809 at 1676289613873Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=DISABLED ttl=0 } at 1676289613885 (+12 ms)Consolidate snapshot: emptySnaptb-1676289612809 at 1676289614349 (+464 ms)Loading Region manifests for emptySnaptb-1676289612809 at 1676289614349Writing data manifest for emptySnaptb-1676289612809 at 1676289614354 (+5 ms)Verifying snapshot: emptySnaptb-1676289612809 at 1676289614367 (+13 ms)Snapshot emptySnaptb-1676289612809 of table testCorruptedSnapshot-1676289612809 completed at 1676289614389 (+22 ms) 2023-02-13 12:00:14,392 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE 2023-02-13 12:00:14,393 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE in 522 msec 2023-02-13 12:00:14,626 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:00:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } is done 2023-02-13 12:00:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:00:14,628 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676289612809 2023-02-13 12:00:14,628 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testCorruptedSnapshot-1676289612809 2023-02-13 12:00:14,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:14,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 12:00:14,633 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676289612809 2023-02-13 12:00:14,633 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289614633"}]},"ts":"1676289614633"} 2023-02-13 12:00:14,635 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=ENABLING in hbase:meta 2023-02-13 12:00:14,675 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:00:14,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN}] 2023-02-13 12:00:14,678 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN 2023-02-13 12:00:14,678 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN 2023-02-13 12:00:14,678 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN 2023-02-13 12:00:14,678 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN 2023-02-13 12:00:14,679 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:14,679 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 12:00:14,680 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:14,680 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:14,680 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 12:00:14,680 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:14,680 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 12:00:14,681 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 12:00:14,681 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN 2023-02-13 12:00:14,682 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN 2023-02-13 12:00:14,682 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:14,682 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:14,682 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 12:00:14,682 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 12:00:14,830 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:00:14,830 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:14,830 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:14,830 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:14,830 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289614830"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289614830"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289614830"}]},"ts":"1676289614830"} 2023-02-13 12:00:14,830 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289614830"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289614830"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289614830"}]},"ts":"1676289614830"} 2023-02-13 12:00:14,830 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:14,830 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:14,831 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289614830"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289614830"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289614830"}]},"ts":"1676289614830"} 2023-02-13 12:00:14,830 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289614830"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289614830"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289614830"}]},"ts":"1676289614830"} 2023-02-13 12:00:14,831 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289614830"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289614830"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289614830"}]},"ts":"1676289614830"} 2023-02-13 12:00:14,832 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=230, state=RUNNABLE; OpenRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:14,832 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=234, state=RUNNABLE; OpenRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:14,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=231, state=RUNNABLE; OpenRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:14,834 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:14,834 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=235, state=RUNNABLE; OpenRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:14,834 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289614834"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289614834"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289614834"}]},"ts":"1676289614834"} 2023-02-13 12:00:14,835 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=233, state=RUNNABLE; OpenRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:14,836 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=232, state=RUNNABLE; OpenRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:14,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 12:00:14,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:14,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:14,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 643a6d6a4f0efd58319e8e38b8b918fb, NAME => 'testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:00:14,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a6a2c6c5a450fab5269836430582d6, NAME => 'testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:00:14,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:14,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:14,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:14,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:14,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:14,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:14,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:14,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:14,994 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:14,994 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:14,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:14,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a0478ecfb8560aee2a65d010b27747ca, NAME => 'testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:00:14,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:14,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:14,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:14,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:14,995 DEBUG [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf 2023-02-13 12:00:14,995 DEBUG [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf 2023-02-13 12:00:14,995 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 643a6d6a4f0efd58319e8e38b8b918fb columnFamilyName cf 2023-02-13 12:00:14,995 DEBUG [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf 2023-02-13 12:00:14,995 DEBUG [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf 2023-02-13 12:00:14,996 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:14,996 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a6a2c6c5a450fab5269836430582d6 columnFamilyName cf 2023-02-13 12:00:14,996 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] regionserver.HStore(310): Store=643a6d6a4f0efd58319e8e38b8b918fb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:14,996 DEBUG [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf 2023-02-13 12:00:14,996 DEBUG [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf 2023-02-13 12:00:14,997 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] regionserver.HStore(310): Store=81a6a2c6c5a450fab5269836430582d6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:14,997 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a0478ecfb8560aee2a65d010b27747ca columnFamilyName cf 2023-02-13 12:00:14,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:14,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:14,998 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] regionserver.HStore(310): Store=a0478ecfb8560aee2a65d010b27747ca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:14,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:14,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:14,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:15,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:15,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:15,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:15,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 643a6d6a4f0efd58319e8e38b8b918fb; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60942295, jitterRate=-0.09188903868198395}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:15,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 643a6d6a4f0efd58319e8e38b8b918fb: Running coprocessor pre-open hook at 1676289614993Writing region info on filesystem at 1676289614993Initializing all the Stores at 1676289614994 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289614994Cleaning up temporary data from old regions at 1676289614999 (+5 ms)Cleaning up detritus from prior splits at 1676289614999Running coprocessor post-open hooks at 1676289615001 (+2 ms)Region opened successfully at 1676289615001 2023-02-13 12:00:15,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a6a2c6c5a450fab5269836430582d6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73901378, jitterRate=0.10121634602546692}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:15,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a6a2c6c5a450fab5269836430582d6: Running coprocessor pre-open hook at 1676289614993Writing region info on filesystem at 1676289614993Initializing all the Stores at 1676289614994 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289614994Cleaning up temporary data from old regions at 1676289614999 (+5 ms)Cleaning up detritus from prior splits at 1676289614999Running coprocessor post-open hooks at 1676289615002 (+3 ms)Region opened successfully at 1676289615002 2023-02-13 12:00:15,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:15,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb., pid=237, masterSystemTime=1676289614983 2023-02-13 12:00:15,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6., pid=239, masterSystemTime=1676289614983 2023-02-13 12:00:15,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:15,003 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:15,003 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:15,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bdb004dd508672ca8f98b4ad3d79f3a, NAME => 'testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:00:15,004 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:15,004 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615004"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289615004"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289615004"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289615004"}]},"ts":"1676289615004"} 2023-02-13 12:00:15,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:15,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:15,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a0478ecfb8560aee2a65d010b27747ca; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70272099, jitterRate=0.04713587462902069}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:15,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:15,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a0478ecfb8560aee2a65d010b27747ca: Running coprocessor pre-open hook at 1676289614995Writing region info on filesystem at 1676289614995Initializing all the Stores at 1676289614995Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289614995Cleaning up temporary data from old regions at 1676289615000 (+5 ms)Cleaning up detritus from prior splits at 1676289615000Running coprocessor post-open hooks at 1676289615004 (+4 ms)Region opened successfully at 1676289615004 2023-02-13 12:00:15,004 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:15,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,004 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289615004"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289615004"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289615004"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289615004"}]},"ts":"1676289615004"} 2023-02-13 12:00:15,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f5abf54abb980cb8f138bde5b646e3c0, NAME => 'testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:00:15,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:15,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca., pid=240, masterSystemTime=1676289614990 2023-02-13 12:00:15,006 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:15,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:15,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:15,006 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 48d6107ef6392fc0224ecd69dad7d2f3, NAME => 'testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:00:15,007 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:15,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,007 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615007"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289615007"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289615007"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289615007"}]},"ts":"1676289615007"} 2023-02-13 12:00:15,007 DEBUG [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf 2023-02-13 12:00:15,007 DEBUG [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf 2023-02-13 12:00:15,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:15,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,007 DEBUG [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf 2023-02-13 12:00:15,007 DEBUG [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf 2023-02-13 12:00:15,007 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f5abf54abb980cb8f138bde5b646e3c0 columnFamilyName cf 2023-02-13 12:00:15,007 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bdb004dd508672ca8f98b4ad3d79f3a columnFamilyName cf 2023-02-13 12:00:15,008 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,008 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] regionserver.HStore(310): Store=f5abf54abb980cb8f138bde5b646e3c0/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:15,008 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] regionserver.HStore(310): Store=9bdb004dd508672ca8f98b4ad3d79f3a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:15,009 DEBUG [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf 2023-02-13 12:00:15,009 DEBUG [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf 2023-02-13 12:00:15,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,009 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 48d6107ef6392fc0224ecd69dad7d2f3 columnFamilyName cf 2023-02-13 12:00:15,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=234 2023-02-13 12:00:15,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=234, state=SUCCESS; OpenRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427 in 176 msec 2023-02-13 12:00:15,012 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=235 2023-02-13 12:00:15,012 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=235, state=SUCCESS; OpenRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478 in 175 msec 2023-02-13 12:00:15,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=233 2023-02-13 12:00:15,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=233, state=SUCCESS; OpenRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518 in 175 msec 2023-02-13 12:00:15,012 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN in 335 msec 2023-02-13 12:00:15,013 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] regionserver.HStore(310): Store=48d6107ef6392fc0224ecd69dad7d2f3/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:15,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,015 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN in 336 msec 2023-02-13 12:00:15,015 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN in 336 msec 2023-02-13 12:00:15,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 9bdb004dd508672ca8f98b4ad3d79f3a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63238963, jitterRate=-0.05766601860523224}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:15,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 9bdb004dd508672ca8f98b4ad3d79f3a: Running coprocessor pre-open hook at 1676289615004Writing region info on filesystem at 1676289615004Initializing all the Stores at 1676289615006 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289615006Cleaning up temporary data from old regions at 1676289615013 (+7 ms)Cleaning up detritus from prior splits at 1676289615014 (+1 ms)Running coprocessor post-open hooks at 1676289615016 (+2 ms)Region opened successfully at 1676289615016 2023-02-13 12:00:15,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,017 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a., pid=238, masterSystemTime=1676289614983 2023-02-13 12:00:15,017 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 48d6107ef6392fc0224ecd69dad7d2f3; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66966039, jitterRate=-0.0021282583475112915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:15,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 48d6107ef6392fc0224ecd69dad7d2f3: Running coprocessor pre-open hook at 1676289615007Writing region info on filesystem at 1676289615007Initializing all the Stores at 1676289615007Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289615008 (+1 ms)Cleaning up temporary data from old regions at 1676289615015 (+7 ms)Cleaning up detritus from prior splits at 1676289615015Running coprocessor post-open hooks at 1676289615017 (+2 ms)Region opened successfully at 1676289615017 2023-02-13 12:00:15,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3., pid=241, masterSystemTime=1676289614990 2023-02-13 12:00:15,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:15,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:15,018 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:15,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened f5abf54abb980cb8f138bde5b646e3c0; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67635266, jitterRate=0.007844001054763794}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:15,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for f5abf54abb980cb8f138bde5b646e3c0: Running coprocessor pre-open hook at 1676289615005Writing region info on filesystem at 1676289615005Initializing all the Stores at 1676289615006 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289615006Cleaning up temporary data from old regions at 1676289615016 (+10 ms)Cleaning up detritus from prior splits at 1676289615016Running coprocessor post-open hooks at 1676289615018 (+2 ms)Region opened successfully at 1676289615018 2023-02-13 12:00:15,019 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615018"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289615018"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289615018"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289615018"}]},"ts":"1676289615018"} 2023-02-13 12:00:15,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0., pid=236, masterSystemTime=1676289614983 2023-02-13 12:00:15,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:15,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:15,020 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:15,020 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615020"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289615020"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289615020"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289615020"}]},"ts":"1676289615020"} 2023-02-13 12:00:15,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,021 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=231 2023-02-13 12:00:15,021 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:15,021 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=231, state=SUCCESS; OpenRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 187 msec 2023-02-13 12:00:15,021 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289615021"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289615021"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289615021"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289615021"}]},"ts":"1676289615021"} 2023-02-13 12:00:15,023 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN in 345 msec 2023-02-13 12:00:15,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=232 2023-02-13 12:00:15,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=232, state=SUCCESS; OpenRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518 in 186 msec 2023-02-13 12:00:15,024 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=230 2023-02-13 12:00:15,024 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=230, state=SUCCESS; OpenRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478 in 191 msec 2023-02-13 12:00:15,024 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN in 347 msec 2023-02-13 12:00:15,025 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=230, resume processing ppid=229 2023-02-13 12:00:15,025 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN in 348 msec 2023-02-13 12:00:15,026 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289615026"}]},"ts":"1676289615026"} 2023-02-13 12:00:15,027 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=ENABLED in hbase:meta 2023-02-13 12:00:15,035 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676289612809' was successfully enabled. 2023-02-13 12:00:15,037 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676289612809 in 406 msec 2023-02-13 12:00:15,337 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676289612809' 2023-02-13 12:00:15,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 12:00:15,386 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676289612809, procId: 229 completed 2023-02-13 12:00:15,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:15,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:15,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:15,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:15,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:15,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:15,407 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676289612809 2023-02-13 12:00:15,407 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,408 INFO [Listener at localhost.localdomain/36409] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:00:15,422 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:00:15,424 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:00:15,425 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:00:15,427 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:00:15,429 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:00:15,436 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676289612809 2023-02-13 12:00:15,437 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testCorruptedSnapshot-1676289612809 2023-02-13 12:00:15,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:15,442 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289615441"}]},"ts":"1676289615441"} 2023-02-13 12:00:15,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 12:00:15,443 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=DISABLING in hbase:meta 2023-02-13 12:00:15,451 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676289612809 to state=DISABLING 2023-02-13 12:00:15,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN}] 2023-02-13 12:00:15,453 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN 2023-02-13 12:00:15,453 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN 2023-02-13 12:00:15,453 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN 2023-02-13 12:00:15,454 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN 2023-02-13 12:00:15,454 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN 2023-02-13 12:00:15,454 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:15,454 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:15,454 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615454"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289615454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289615454"}]},"ts":"1676289615454"} 2023-02-13 12:00:15,454 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:15,454 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:15,454 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289615454"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289615454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289615454"}]},"ts":"1676289615454"} 2023-02-13 12:00:15,455 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615454"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289615454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289615454"}]},"ts":"1676289615454"} 2023-02-13 12:00:15,454 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:15,455 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615454"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289615454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289615454"}]},"ts":"1676289615454"} 2023-02-13 12:00:15,454 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615454"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289615454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289615454"}]},"ts":"1676289615454"} 2023-02-13 12:00:15,455 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=247, state=RUNNABLE; CloseRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:15,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=248, state=RUNNABLE; CloseRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:15,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=244, state=RUNNABLE; CloseRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:15,458 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=245, state=RUNNABLE; CloseRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:15,458 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=246, state=RUNNABLE; CloseRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:15,460 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN 2023-02-13 12:00:15,460 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:15,460 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289615460"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289615460"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289615460"}]},"ts":"1676289615460"} 2023-02-13 12:00:15,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=243, state=RUNNABLE; CloseRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:15,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:15,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 643a6d6a4f0efd58319e8e38b8b918fb, disabling compactions & flushes 2023-02-13 12:00:15,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:15,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:15,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. after waiting 0 ms 2023-02-13 12:00:15,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:15,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 643a6d6a4f0efd58319e8e38b8b918fb 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 12:00:15,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:15,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:15,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a6a2c6c5a450fab5269836430582d6, disabling compactions & flushes 2023-02-13 12:00:15,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a0478ecfb8560aee2a65d010b27747ca, disabling compactions & flushes 2023-02-13 12:00:15,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:15,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:15,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. after waiting 0 ms 2023-02-13 12:00:15,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:15,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:15,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing a0478ecfb8560aee2a65d010b27747ca 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 12:00:15,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:15,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. after waiting 0 ms 2023-02-13 12:00:15,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:15,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 81a6a2c6c5a450fab5269836430582d6 1/1 column families, dataSize=20.93 KB heapSize=45.25 KB 2023-02-13 12:00:15,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021348d3eb972f194efdbeddfeaaf53481ba to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d52023021348d3eb972f194efdbeddfeaaf53481ba 2023-02-13 12:00:15,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d9f7e3c6e703460ea1c207b0f8651869 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d9f7e3c6e703460ea1c207b0f8651869 2023-02-13 12:00:15,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302131dd1c12e26a340a6acf0fc609d79dace to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302131dd1c12e26a340a6acf0fc609d79dace 2023-02-13 12:00:15,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=20.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/.tmp/cf/980a841c48a64c50ac0f46aa4f34b1fb 2023-02-13 12:00:15,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/.tmp/cf/980a841c48a64c50ac0f46aa4f34b1fb as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/980a841c48a64c50ac0f46aa4f34b1fb 2023-02-13 12:00:15,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/.tmp/cf/2ce8d42bb8dc4433ab86bdffdc44b465 2023-02-13 12:00:15,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/980a841c48a64c50ac0f46aa4f34b1fb, entries=320, sequenceid=8, filesize=60.1 K 2023-02-13 12:00:15,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~20.93 KB/21436, heapSize ~45.23 KB/46320, currentSize=0 B/0 for 81a6a2c6c5a450fab5269836430582d6 in 52ms, sequenceid=8, compaction requested=false 2023-02-13 12:00:15,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/.tmp/cf/2ce8d42bb8dc4433ab86bdffdc44b465 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/2ce8d42bb8dc4433ab86bdffdc44b465 2023-02-13 12:00:15,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/.tmp/cf/c2a7ca3b619048038c36f07886ad7fa7 2023-02-13 12:00:15,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:00:15,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:15,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a6a2c6c5a450fab5269836430582d6: Waiting for close lock at 1676289615613Running coprocessor pre-close hooks at 1676289615613Disabling compacts and flushes for region at 1676289615613Disabling writes for close at 1676289615614 (+1 ms)Obtaining lock to block concurrent updates at 1676289615614Preparing flush snapshotting stores in 81a6a2c6c5a450fab5269836430582d6 at 1676289615614Finished memstore snapshotting testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6., syncing WAL and waiting on mvcc, flushsize=dataSize=21436, getHeapSize=46320, getOffHeapSize=0, getCellsCount=320 at 1676289615615 (+1 ms)Flushing stores of testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. at 1676289615617 (+2 ms)Flushing 81a6a2c6c5a450fab5269836430582d6/cf: creating writer at 1676289615618 (+1 ms)Flushing mob file 81a6a2c6c5a450fab5269836430582d6/cf: appending metadata at 1676289615624 (+6 ms)Flushing mob file 81a6a2c6c5a450fab5269836430582d6/cf: closing flushed file at 1676289615624Flushing 81a6a2c6c5a450fab5269836430582d6/cf: appending metadata at 1676289615648 (+24 ms)Flushing 81a6a2c6c5a450fab5269836430582d6/cf: closing flushed file at 1676289615648Flushing 81a6a2c6c5a450fab5269836430582d6/cf: reopening flushed file at 1676289615662 (+14 ms)Finished flush of dataSize ~20.93 KB/21436, heapSize ~45.23 KB/46320, currentSize=0 B/0 for 81a6a2c6c5a450fab5269836430582d6 in 52ms, sequenceid=8, compaction requested=false at 1676289615666 (+4 ms)Writing region close event to WAL at 1676289615667 (+1 ms)Running coprocessor post-close hooks at 1676289615670 (+3 ms)Closed at 1676289615670 2023-02-13 12:00:15,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/2ce8d42bb8dc4433ab86bdffdc44b465, entries=33, sequenceid=8, filesize=10.6 K 2023-02-13 12:00:15,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:15,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing f5abf54abb980cb8f138bde5b646e3c0, disabling compactions & flushes 2023-02-13 12:00:15,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/.tmp/cf/c2a7ca3b619048038c36f07886ad7fa7 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/c2a7ca3b619048038c36f07886ad7fa7 2023-02-13 12:00:15,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. after waiting 0 ms 2023-02-13 12:00:15,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 643a6d6a4f0efd58319e8e38b8b918fb in 63ms, sequenceid=8, compaction requested=false 2023-02-13 12:00:15,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing f5abf54abb980cb8f138bde5b646e3c0 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-13 12:00:15,672 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=CLOSED 2023-02-13 12:00:15,672 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289615672"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289615672"}]},"ts":"1676289615672"} 2023-02-13 12:00:15,675 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=248 2023-02-13 12:00:15,675 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=248, state=SUCCESS; CloseRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478 in 217 msec 2023-02-13 12:00:15,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:00:15,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:15,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 643a6d6a4f0efd58319e8e38b8b918fb: Waiting for close lock at 1676289615608Running coprocessor pre-close hooks at 1676289615608Disabling compacts and flushes for region at 1676289615608Disabling writes for close at 1676289615609 (+1 ms)Obtaining lock to block concurrent updates at 1676289615609Preparing flush snapshotting stores in 643a6d6a4f0efd58319e8e38b8b918fb at 1676289615609Finished memstore snapshotting testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676289615609Flushing stores of testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. at 1676289615610 (+1 ms)Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: creating writer at 1676289615612 (+2 ms)Flushing mob file 643a6d6a4f0efd58319e8e38b8b918fb/cf: appending metadata at 1676289615622 (+10 ms)Flushing mob file 643a6d6a4f0efd58319e8e38b8b918fb/cf: closing flushed file at 1676289615622Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: appending metadata at 1676289615655 (+33 ms)Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: closing flushed file at 1676289615655Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: reopening flushed file at 1676289615669 (+14 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 643a6d6a4f0efd58319e8e38b8b918fb in 63ms, sequenceid=8, compaction requested=false at 1676289615672 (+3 ms)Writing region close event to WAL at 1676289615672Running coprocessor post-close hooks at 1676289615676 (+4 ms)Closed at 1676289615676 2023-02-13 12:00:15,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/c2a7ca3b619048038c36f07886ad7fa7, entries=36, sequenceid=8, filesize=11.1 K 2023-02-13 12:00:15,677 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN in 223 msec 2023-02-13 12:00:15,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for a0478ecfb8560aee2a65d010b27747ca in 63ms, sequenceid=8, compaction requested=false 2023-02-13 12:00:15,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:15,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 9bdb004dd508672ca8f98b4ad3d79f3a, disabling compactions & flushes 2023-02-13 12:00:15,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:15,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:15,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. after waiting 0 ms 2023-02-13 12:00:15,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:15,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 9bdb004dd508672ca8f98b4ad3d79f3a 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-13 12:00:15,678 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=CLOSED 2023-02-13 12:00:15,678 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615678"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289615678"}]},"ts":"1676289615678"} 2023-02-13 12:00:15,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=247 2023-02-13 12:00:15,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=247, state=SUCCESS; CloseRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427 in 224 msec 2023-02-13 12:00:15,683 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN in 230 msec 2023-02-13 12:00:15,684 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:00:15,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:15,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a0478ecfb8560aee2a65d010b27747ca: Waiting for close lock at 1676289615613Running coprocessor pre-close hooks at 1676289615613Disabling compacts and flushes for region at 1676289615613Disabling writes for close at 1676289615614 (+1 ms)Obtaining lock to block concurrent updates at 1676289615614Preparing flush snapshotting stores in a0478ecfb8560aee2a65d010b27747ca at 1676289615614Finished memstore snapshotting testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676289615614Flushing stores of testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. at 1676289615615 (+1 ms)Flushing a0478ecfb8560aee2a65d010b27747ca/cf: creating writer at 1676289615616 (+1 ms)Flushing mob file a0478ecfb8560aee2a65d010b27747ca/cf: appending metadata at 1676289615625 (+9 ms)Flushing mob file a0478ecfb8560aee2a65d010b27747ca/cf: closing flushed file at 1676289615625Flushing a0478ecfb8560aee2a65d010b27747ca/cf: appending metadata at 1676289615653 (+28 ms)Flushing a0478ecfb8560aee2a65d010b27747ca/cf: closing flushed file at 1676289615653Flushing a0478ecfb8560aee2a65d010b27747ca/cf: reopening flushed file at 1676289615672 (+19 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for a0478ecfb8560aee2a65d010b27747ca in 63ms, sequenceid=8, compaction requested=false at 1676289615677 (+5 ms)Writing region close event to WAL at 1676289615682 (+5 ms)Running coprocessor post-close hooks at 1676289615684 (+2 ms)Closed at 1676289615684 2023-02-13 12:00:15,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:15,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 48d6107ef6392fc0224ecd69dad7d2f3, disabling compactions & flushes 2023-02-13 12:00:15,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:15,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:15,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. after waiting 0 ms 2023-02-13 12:00:15,686 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=CLOSED 2023-02-13 12:00:15,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:15,686 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615686"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289615686"}]},"ts":"1676289615686"} 2023-02-13 12:00:15,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 48d6107ef6392fc0224ecd69dad7d2f3 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-13 12:00:15,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=246 2023-02-13 12:00:15,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=246, state=SUCCESS; CloseRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518 in 229 msec 2023-02-13 12:00:15,692 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN in 238 msec 2023-02-13 12:00:15,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 12:00:15,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213829051f01ee2458ab84a8d40fba682a7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213829051f01ee2458ab84a8d40fba682a7 2023-02-13 12:00:15,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302138d0ecd628abb4f82ac488c033938186b to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b202302138d0ecd628abb4f82ac488c033938186b 2023-02-13 12:00:15,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021357f9f893317244efb84943eb9cb531b7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c2023021357f9f893317244efb84943eb9cb531b7 2023-02-13 12:00:15,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/.tmp/cf/53ceb530421c4eb1b807a459c580ec75 2023-02-13 12:00:15,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/.tmp/cf/53ceb530421c4eb1b807a459c580ec75 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/53ceb530421c4eb1b807a459c580ec75 2023-02-13 12:00:15,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/.tmp/cf/f989a8e47ab847c8a8e2f4de64587c29 2023-02-13 12:00:15,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/.tmp/cf/fb4444c916444f24b29bfd7c0a4319e3 2023-02-13 12:00:15,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/53ceb530421c4eb1b807a459c580ec75, entries=38, sequenceid=8, filesize=11.4 K 2023-02-13 12:00:15,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for f5abf54abb980cb8f138bde5b646e3c0 in 40ms, sequenceid=8, compaction requested=false 2023-02-13 12:00:15,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/.tmp/cf/f989a8e47ab847c8a8e2f4de64587c29 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/f989a8e47ab847c8a8e2f4de64587c29 2023-02-13 12:00:15,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/.tmp/cf/fb4444c916444f24b29bfd7c0a4319e3 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3 2023-02-13 12:00:15,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:00:15,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:15,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for f5abf54abb980cb8f138bde5b646e3c0: Waiting for close lock at 1676289615671Running coprocessor pre-close hooks at 1676289615671Disabling compacts and flushes for region at 1676289615671Disabling writes for close at 1676289615672 (+1 ms)Obtaining lock to block concurrent updates at 1676289615672Preparing flush snapshotting stores in f5abf54abb980cb8f138bde5b646e3c0 at 1676289615672Finished memstore snapshotting testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676289615672Flushing stores of testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. at 1676289615672Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: creating writer at 1676289615673 (+1 ms)Flushing mob file f5abf54abb980cb8f138bde5b646e3c0/cf: appending metadata at 1676289615677 (+4 ms)Flushing mob file f5abf54abb980cb8f138bde5b646e3c0/cf: closing flushed file at 1676289615677Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: appending metadata at 1676289615695 (+18 ms)Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: closing flushed file at 1676289615695Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: reopening flushed file at 1676289615709 (+14 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for f5abf54abb980cb8f138bde5b646e3c0 in 40ms, sequenceid=8, compaction requested=false at 1676289615712 (+3 ms)Writing region close event to WAL at 1676289615713 (+1 ms)Running coprocessor post-close hooks at 1676289615715 (+2 ms)Closed at 1676289615715 2023-02-13 12:00:15,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/f989a8e47ab847c8a8e2f4de64587c29, entries=39, sequenceid=8, filesize=11.6 K 2023-02-13 12:00:15,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3, entries=34, sequenceid=8, filesize=10.8 K 2023-02-13 12:00:15,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 9bdb004dd508672ca8f98b4ad3d79f3a in 39ms, sequenceid=8, compaction requested=false 2023-02-13 12:00:15,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:15,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 48d6107ef6392fc0224ecd69dad7d2f3 in 30ms, sequenceid=8, compaction requested=false 2023-02-13 12:00:15,716 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=CLOSED 2023-02-13 12:00:15,716 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289615716"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289615716"}]},"ts":"1676289615716"} 2023-02-13 12:00:15,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:00:15,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:00:15,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:15,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 48d6107ef6392fc0224ecd69dad7d2f3: Waiting for close lock at 1676289615686Running coprocessor pre-close hooks at 1676289615686Disabling compacts and flushes for region at 1676289615686Disabling writes for close at 1676289615686Obtaining lock to block concurrent updates at 1676289615686Preparing flush snapshotting stores in 48d6107ef6392fc0224ecd69dad7d2f3 at 1676289615686Finished memstore snapshotting testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676289615686Flushing stores of testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. at 1676289615687 (+1 ms)Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: creating writer at 1676289615687Flushing mob file 48d6107ef6392fc0224ecd69dad7d2f3/cf: appending metadata at 1676289615691 (+4 ms)Flushing mob file 48d6107ef6392fc0224ecd69dad7d2f3/cf: closing flushed file at 1676289615691Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: appending metadata at 1676289615704 (+13 ms)Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: closing flushed file at 1676289615704Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: reopening flushed file at 1676289615713 (+9 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 48d6107ef6392fc0224ecd69dad7d2f3 in 30ms, sequenceid=8, compaction requested=false at 1676289615716 (+3 ms)Writing region close event to WAL at 1676289615717 (+1 ms)Running coprocessor post-close hooks at 1676289615719 (+2 ms)Closed at 1676289615719 2023-02-13 12:00:15,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=243 2023-02-13 12:00:15,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=243, state=SUCCESS; CloseRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478 in 254 msec 2023-02-13 12:00:15,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:15,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 9bdb004dd508672ca8f98b4ad3d79f3a: Waiting for close lock at 1676289615677Running coprocessor pre-close hooks at 1676289615677Disabling compacts and flushes for region at 1676289615677Disabling writes for close at 1676289615677Obtaining lock to block concurrent updates at 1676289615678 (+1 ms)Preparing flush snapshotting stores in 9bdb004dd508672ca8f98b4ad3d79f3a at 1676289615678Finished memstore snapshotting testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676289615678Flushing stores of testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. at 1676289615678Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: creating writer at 1676289615681 (+3 ms)Flushing mob file 9bdb004dd508672ca8f98b4ad3d79f3a/cf: appending metadata at 1676289615685 (+4 ms)Flushing mob file 9bdb004dd508672ca8f98b4ad3d79f3a/cf: closing flushed file at 1676289615685Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: appending metadata at 1676289615703 (+18 ms)Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: closing flushed file at 1676289615704 (+1 ms)Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: reopening flushed file at 1676289615713 (+9 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 9bdb004dd508672ca8f98b4ad3d79f3a in 39ms, sequenceid=8, compaction requested=false at 1676289615716 (+3 ms)Writing region close event to WAL at 1676289615716Running coprocessor post-close hooks at 1676289615719 (+3 ms)Closed at 1676289615719 2023-02-13 12:00:15,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:15,720 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN in 267 msec 2023-02-13 12:00:15,720 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=CLOSED 2023-02-13 12:00:15,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:15,720 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289615720"}]},"ts":"1676289615720"} 2023-02-13 12:00:15,721 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=CLOSED 2023-02-13 12:00:15,721 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289615721"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289615721"}]},"ts":"1676289615721"} 2023-02-13 12:00:15,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=245 2023-02-13 12:00:15,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=245, state=SUCCESS; CloseRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518 in 263 msec 2023-02-13 12:00:15,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=244 2023-02-13 12:00:15,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=244, state=SUCCESS; CloseRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 265 msec 2023-02-13 12:00:15,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN in 271 msec 2023-02-13 12:00:15,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=244, resume processing ppid=242 2023-02-13 12:00:15,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN in 271 msec 2023-02-13 12:00:15,725 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289615725"}]},"ts":"1676289615725"} 2023-02-13 12:00:15,726 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=DISABLED in hbase:meta 2023-02-13 12:00:15,735 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676289612809 to state=DISABLED 2023-02-13 12:00:15,736 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676289612809 in 298 msec 2023-02-13 12:00:16,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 12:00:16,195 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676289612809, procId: 242 completed 2023-02-13 12:00:16,195 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } 2023-02-13 12:00:16,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676289616195 (current time:1676289616195). 2023-02-13 12:00:16,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:00:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(139): Connect 0x102fe3b0 to 127.0.0.1:63172 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:00:16,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fa42e52, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:00:16,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:00:16,214 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52164, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:00:16,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x102fe3b0 to 127.0.0.1:63172 2023-02-13 12:00:16,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:16,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:00:16,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } 2023-02-13 12:00:16,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE 2023-02-13 12:00:16,219 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE 2023-02-13 12:00:16,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } 2023-02-13 12:00:16,219 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE; timeout=600000, timestamp=1676290216219 2023-02-13 12:00:16,219 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676289612809 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676289612809 2023-02-13 12:00:16,220 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:00:16,220 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:00:16,228 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=DISABLED ttl=0 } 2023-02-13 12:00:16,231 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:16,231 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:16,231 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:16,232 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:16,232 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:16,232 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:16,232 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:16,232 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:16,233 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:16,233 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:16,233 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:16,233 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:16,234 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:00:16,234 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:00:16,236 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/2ce8d42bb8dc4433ab86bdffdc44b465] hfiles 2023-02-13 12:00:16,236 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/53ceb530421c4eb1b807a459c580ec75] hfiles 2023-02-13 12:00:16,236 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/2ce8d42bb8dc4433ab86bdffdc44b465 2023-02-13 12:00:16,236 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/f989a8e47ab847c8a8e2f4de64587c29] hfiles 2023-02-13 12:00:16,236 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/53ceb530421c4eb1b807a459c580ec75 2023-02-13 12:00:16,236 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/f989a8e47ab847c8a8e2f4de64587c29 2023-02-13 12:00:16,240 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3] hfiles 2023-02-13 12:00:16,240 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3 2023-02-13 12:00:16,241 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/980a841c48a64c50ac0f46aa4f34b1fb] hfiles 2023-02-13 12:00:16,241 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/980a841c48a64c50ac0f46aa4f34b1fb 2023-02-13 12:00:16,242 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/c2a7ca3b619048038c36f07886ad7fa7] hfiles 2023-02-13 12:00:16,242 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/c2a7ca3b619048038c36f07886ad7fa7 2023-02-13 12:00:16,249 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302131dd1c12e26a340a6acf0fc609d79dace, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b202302138d0ecd628abb4f82ac488c033938186b, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c2023021357f9f893317244efb84943eb9cb531b7, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213829051f01ee2458ab84a8d40fba682a7, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d52023021348d3eb972f194efdbeddfeaaf53481ba, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d9f7e3c6e703460ea1c207b0f8651869] hfiles 2023-02-13 12:00:16,249 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302131dd1c12e26a340a6acf0fc609d79dace 2023-02-13 12:00:16,249 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b202302138d0ecd628abb4f82ac488c033938186b 2023-02-13 12:00:16,249 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c2023021357f9f893317244efb84943eb9cb531b7 2023-02-13 12:00:16,249 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213829051f01ee2458ab84a8d40fba682a7 2023-02-13 12:00:16,249 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d52023021348d3eb972f194efdbeddfeaaf53481ba 2023-02-13 12:00:16,250 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d9f7e3c6e703460ea1c207b0f8651869 2023-02-13 12:00:16,267 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=DISABLED ttl=0 } as finished. 2023-02-13 12:00:16,267 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676289612809 2023-02-13 12:00:16,267 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289612809 2023-02-13 12:00:16,283 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289612809 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/snaptb0-1676289612809 2023-02-13 12:00:16,303 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676289612809 of table testCorruptedSnapshot-1676289612809 completed 2023-02-13 12:00:16,303 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289612809 2023-02-13 12:00:16,303 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.hbase-snapshot/.tmp/snaptb0-1676289612809 2023-02-13 12:00:16,303 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676289612809 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676289612809 at 1676289616219Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=DISABLED ttl=0 } at 1676289616229 (+10 ms)Consolidate snapshot: snaptb0-1676289612809 at 1676289616267 (+38 ms)Loading Region manifests for snaptb0-1676289612809 at 1676289616267Writing data manifest for snaptb0-1676289612809 at 1676289616270 (+3 ms)Verifying snapshot: snaptb0-1676289612809 at 1676289616276 (+6 ms)Snapshot snaptb0-1676289612809 of table testCorruptedSnapshot-1676289612809 completed at 1676289616303 (+27 ms) 2023-02-13 12:00:16,304 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE 2023-02-13 12:00:16,305 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676289612809, type=EXCLUSIVE in 87 msec 2023-02-13 12:00:16,470 DEBUG [Listener at localhost.localdomain/36409] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:00:16,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 } is done 2023-02-13 12:00:16,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676289612809 table=testCorruptedSnapshot-1676289612809 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:00:16,474 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676289612809 2023-02-13 12:00:16,475 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testCorruptedSnapshot-1676289612809 2023-02-13 12:00:16,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:16,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 12:00:16,478 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676289612809 2023-02-13 12:00:16,479 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289616479"}]},"ts":"1676289616479"} 2023-02-13 12:00:16,480 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=ENABLING in hbase:meta 2023-02-13 12:00:16,491 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:00:16,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN}] 2023-02-13 12:00:16,492 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN 2023-02-13 12:00:16,492 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN 2023-02-13 12:00:16,492 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN 2023-02-13 12:00:16,492 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN 2023-02-13 12:00:16,493 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:16,493 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:16,493 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 12:00:16,493 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:16,493 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 12:00:16,493 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 12:00:16,493 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:16,493 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34233,1676289577518; forceNewPlan=false, retain=true 2023-02-13 12:00:16,493 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN 2023-02-13 12:00:16,494 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN 2023-02-13 12:00:16,494 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:16,494 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,45135,1676289577478; forceNewPlan=false, retain=true 2023-02-13 12:00:16,495 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:16,495 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,35305,1676289577427; forceNewPlan=false, retain=true 2023-02-13 12:00:16,643 INFO [jenkins-hbase16:39205] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:00:16,644 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:16,644 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:16,644 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:16,644 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:16,644 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:16,644 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289616644"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289616644"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289616644"}]},"ts":"1676289616644"} 2023-02-13 12:00:16,644 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616644"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289616644"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289616644"}]},"ts":"1676289616644"} 2023-02-13 12:00:16,644 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616644"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289616644"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289616644"}]},"ts":"1676289616644"} 2023-02-13 12:00:16,644 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289616643"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289616643"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289616643"}]},"ts":"1676289616643"} 2023-02-13 12:00:16,645 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616644"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289616644"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289616644"}]},"ts":"1676289616644"} 2023-02-13 12:00:16,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=257, state=RUNNABLE; OpenRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:16,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=258, state=RUNNABLE; OpenRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:16,650 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=260, state=RUNNABLE; OpenRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:16,651 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=262, state=RUNNABLE; OpenRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:16,653 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:16,653 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616653"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289616653"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289616653"}]},"ts":"1676289616653"} 2023-02-13 12:00:16,653 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=261, state=RUNNABLE; OpenRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:16,654 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=259, state=RUNNABLE; OpenRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:16,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 12:00:16,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:16,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:16,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f5abf54abb980cb8f138bde5b646e3c0, NAME => 'testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:00:16,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bdb004dd508672ca8f98b4ad3d79f3a, NAME => 'testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:00:16,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:16,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:16,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:16,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 48d6107ef6392fc0224ecd69dad7d2f3, NAME => 'testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:16,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:16,805 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:16,805 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:16,805 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:16,805 DEBUG [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf 2023-02-13 12:00:16,805 DEBUG [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf 2023-02-13 12:00:16,806 DEBUG [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf 2023-02-13 12:00:16,806 DEBUG [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf 2023-02-13 12:00:16,806 DEBUG [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf 2023-02-13 12:00:16,806 DEBUG [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf 2023-02-13 12:00:16,806 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f5abf54abb980cb8f138bde5b646e3c0 columnFamilyName cf 2023-02-13 12:00:16,806 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 48d6107ef6392fc0224ecd69dad7d2f3 columnFamilyName cf 2023-02-13 12:00:16,806 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bdb004dd508672ca8f98b4ad3d79f3a columnFamilyName cf 2023-02-13 12:00:16,812 DEBUG [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3 2023-02-13 12:00:16,812 DEBUG [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/53ceb530421c4eb1b807a459c580ec75 2023-02-13 12:00:16,812 DEBUG [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/f989a8e47ab847c8a8e2f4de64587c29 2023-02-13 12:00:16,812 INFO [StoreOpener-48d6107ef6392fc0224ecd69dad7d2f3-1] regionserver.HStore(310): Store=48d6107ef6392fc0224ecd69dad7d2f3/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:16,812 INFO [StoreOpener-f5abf54abb980cb8f138bde5b646e3c0-1] regionserver.HStore(310): Store=f5abf54abb980cb8f138bde5b646e3c0/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:16,812 INFO [StoreOpener-9bdb004dd508672ca8f98b4ad3d79f3a-1] regionserver.HStore(310): Store=9bdb004dd508672ca8f98b4ad3d79f3a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:16,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:16,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:16,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:16,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:16,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:16,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:16,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:16,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:16,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 48d6107ef6392fc0224ecd69dad7d2f3; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61295837, jitterRate=-0.08662085235118866}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:16,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 9bdb004dd508672ca8f98b4ad3d79f3a; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67802329, jitterRate=0.010333433747291565}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:16,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 48d6107ef6392fc0224ecd69dad7d2f3: Running coprocessor pre-open hook at 1676289616804Writing region info on filesystem at 1676289616804Initializing all the Stores at 1676289616805 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289616805Cleaning up temporary data from old regions at 1676289616814 (+9 ms)Cleaning up detritus from prior splits at 1676289616815 (+1 ms)Running coprocessor post-open hooks at 1676289616817 (+2 ms)Region opened successfully at 1676289616817 2023-02-13 12:00:16,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 9bdb004dd508672ca8f98b4ad3d79f3a: Running coprocessor pre-open hook at 1676289616804Writing region info on filesystem at 1676289616804Initializing all the Stores at 1676289616804Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289616805 (+1 ms)Cleaning up temporary data from old regions at 1676289616814 (+9 ms)Cleaning up detritus from prior splits at 1676289616815 (+1 ms)Running coprocessor post-open hooks at 1676289616817 (+2 ms)Region opened successfully at 1676289616817 2023-02-13 12:00:16,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3., pid=268, masterSystemTime=1676289616802 2023-02-13 12:00:16,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a., pid=264, masterSystemTime=1676289616801 2023-02-13 12:00:16,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:16,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:16,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:16,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:16,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened f5abf54abb980cb8f138bde5b646e3c0; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63197388, jitterRate=-0.058285534381866455}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:16,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 643a6d6a4f0efd58319e8e38b8b918fb, NAME => 'testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:00:16,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for f5abf54abb980cb8f138bde5b646e3c0: Running coprocessor pre-open hook at 1676289616804Writing region info on filesystem at 1676289616804Initializing all the Stores at 1676289616804Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289616804Cleaning up temporary data from old regions at 1676289616816 (+12 ms)Cleaning up detritus from prior splits at 1676289616817 (+1 ms)Running coprocessor post-open hooks at 1676289616819 (+2 ms)Region opened successfully at 1676289616819 2023-02-13 12:00:16,819 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:16,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:16,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:16,819 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616819"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289616819"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289616819"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289616819"}]},"ts":"1676289616819"} 2023-02-13 12:00:16,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:16,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:16,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:16,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:16,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:16,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0., pid=263, masterSystemTime=1676289616801 2023-02-13 12:00:16,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a0478ecfb8560aee2a65d010b27747ca, NAME => 'testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:00:16,820 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:16,820 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616820"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289616820"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289616820"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289616820"}]},"ts":"1676289616820"} 2023-02-13 12:00:16,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:16,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:16,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:16,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:16,821 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:16,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:16,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:16,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:16,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a6a2c6c5a450fab5269836430582d6, NAME => 'testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:00:16,822 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:16,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676289612809 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:16,822 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289616822"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289616822"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289616822"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289616822"}]},"ts":"1676289616822"} 2023-02-13 12:00:16,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:00:16,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:16,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:16,822 DEBUG [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf 2023-02-13 12:00:16,822 DEBUG [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf 2023-02-13 12:00:16,822 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:16,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=258 2023-02-13 12:00:16,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=258, state=SUCCESS; OpenRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 173 msec 2023-02-13 12:00:16,823 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 643a6d6a4f0efd58319e8e38b8b918fb columnFamilyName cf 2023-02-13 12:00:16,823 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=259 2023-02-13 12:00:16,823 DEBUG [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf 2023-02-13 12:00:16,823 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=259, state=SUCCESS; OpenRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518 in 168 msec 2023-02-13 12:00:16,823 DEBUG [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf 2023-02-13 12:00:16,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, ASSIGN in 332 msec 2023-02-13 12:00:16,824 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a0478ecfb8560aee2a65d010b27747ca columnFamilyName cf 2023-02-13 12:00:16,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, ASSIGN in 332 msec 2023-02-13 12:00:16,825 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=257 2023-02-13 12:00:16,825 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=257, state=SUCCESS; OpenRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478 in 176 msec 2023-02-13 12:00:16,825 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:16,825 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, ASSIGN in 334 msec 2023-02-13 12:00:16,828 DEBUG [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf 2023-02-13 12:00:16,828 DEBUG [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf 2023-02-13 12:00:16,828 DEBUG [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/2ce8d42bb8dc4433ab86bdffdc44b465 2023-02-13 12:00:16,828 INFO [StoreOpener-643a6d6a4f0efd58319e8e38b8b918fb-1] regionserver.HStore(310): Store=643a6d6a4f0efd58319e8e38b8b918fb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:16,828 DEBUG [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/c2a7ca3b619048038c36f07886ad7fa7 2023-02-13 12:00:16,829 INFO [StoreOpener-a0478ecfb8560aee2a65d010b27747ca-1] regionserver.HStore(310): Store=a0478ecfb8560aee2a65d010b27747ca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:16,829 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a6a2c6c5a450fab5269836430582d6 columnFamilyName cf 2023-02-13 12:00:16,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:16,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:16,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:16,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:16,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:16,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:16,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 643a6d6a4f0efd58319e8e38b8b918fb; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70576686, jitterRate=0.05167457461357117}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:16,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened a0478ecfb8560aee2a65d010b27747ca; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61491585, jitterRate=-0.08370397984981537}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:16,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 643a6d6a4f0efd58319e8e38b8b918fb: Running coprocessor pre-open hook at 1676289616820Writing region info on filesystem at 1676289616820Initializing all the Stores at 1676289616820Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289616820Cleaning up temporary data from old regions at 1676289616830 (+10 ms)Cleaning up detritus from prior splits at 1676289616831 (+1 ms)Running coprocessor post-open hooks at 1676289616833 (+2 ms)Region opened successfully at 1676289616833 2023-02-13 12:00:16,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for a0478ecfb8560aee2a65d010b27747ca: Running coprocessor pre-open hook at 1676289616820Writing region info on filesystem at 1676289616820Initializing all the Stores at 1676289616821 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289616821Cleaning up temporary data from old regions at 1676289616830 (+9 ms)Cleaning up detritus from prior splits at 1676289616831 (+1 ms)Running coprocessor post-open hooks at 1676289616833 (+2 ms)Region opened successfully at 1676289616833 2023-02-13 12:00:16,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca., pid=265, masterSystemTime=1676289616802 2023-02-13 12:00:16,834 DEBUG [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/980a841c48a64c50ac0f46aa4f34b1fb 2023-02-13 12:00:16,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb., pid=267, masterSystemTime=1676289616801 2023-02-13 12:00:16,834 INFO [StoreOpener-81a6a2c6c5a450fab5269836430582d6-1] regionserver.HStore(310): Store=81a6a2c6c5a450fab5269836430582d6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:00:16,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:16,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:16,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:16,835 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:16,836 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616835"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289616835"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289616835"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289616835"}]},"ts":"1676289616835"} 2023-02-13 12:00:16,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:16,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:16,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:16,836 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:16,836 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289616836"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289616836"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289616836"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289616836"}]},"ts":"1676289616836"} 2023-02-13 12:00:16,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=261 2023-02-13 12:00:16,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=261, state=SUCCESS; OpenRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427 in 184 msec 2023-02-13 12:00:16,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=260 2023-02-13 12:00:16,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=260, state=SUCCESS; OpenRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518 in 187 msec 2023-02-13 12:00:16,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, ASSIGN in 347 msec 2023-02-13 12:00:16,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:16,839 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, ASSIGN in 347 msec 2023-02-13 12:00:16,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a6a2c6c5a450fab5269836430582d6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67039852, jitterRate=-0.0010283589363098145}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:00:16,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a6a2c6c5a450fab5269836430582d6: Running coprocessor pre-open hook at 1676289616822Writing region info on filesystem at 1676289616822Initializing all the Stores at 1676289616824 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676289616824Cleaning up temporary data from old regions at 1676289616837 (+13 ms)Cleaning up detritus from prior splits at 1676289616837Running coprocessor post-open hooks at 1676289616839 (+2 ms)Region opened successfully at 1676289616839 2023-02-13 12:00:16,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6., pid=266, masterSystemTime=1676289616801 2023-02-13 12:00:16,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:16,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:16,841 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:16,841 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289616841"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676289616841"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676289616841"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676289616841"}]},"ts":"1676289616841"} 2023-02-13 12:00:16,842 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=262 2023-02-13 12:00:16,843 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=262, state=SUCCESS; OpenRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478 in 191 msec 2023-02-13 12:00:16,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=262, resume processing ppid=256 2023-02-13 12:00:16,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, ASSIGN in 351 msec 2023-02-13 12:00:16,844 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289616844"}]},"ts":"1676289616844"} 2023-02-13 12:00:16,845 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=ENABLED in hbase:meta 2023-02-13 12:00:16,857 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676289612809' was successfully enabled. 2023-02-13 12:00:16,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676289612809 in 382 msec 2023-02-13 12:00:17,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 12:00:17,232 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676289612809, procId: 256 completed 2023-02-13 12:00:17,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:17,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:17,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35305] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:17,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34233] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:17,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:17,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45135] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:00:17,250 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676289612809 2023-02-13 12:00:17,250 DEBUG [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:17,250 INFO [Listener at localhost.localdomain/36409] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:00:17,379 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:00:17,412 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:00:17,487 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:00:17,576 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:00:17,661 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:00:17,774 DEBUG [Listener at localhost.localdomain/36409] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:00:18,822 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: fb4444c916444f24b29bfd7c0a4319e3 in the real (hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3) or archive (hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3) directory for the primary table. 2023-02-13 12:00:18,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: fb4444c916444f24b29bfd7c0a4319e3 in the real (hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3) or archive (hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:00:18,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] ipc.CallRunner(144): callId: 374 service: MasterService methodName: RestoreSnapshot size: 114 connection: 188.40.62.62:52694 deadline: 1676289678817, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: fb4444c916444f24b29bfd7c0a4319e3 in the real (hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3) or archive (hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/fb4444c916444f24b29bfd7c0a4319e3) directory for the primary table. 2023-02-13 12:00:18,845 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676289612809 2023-02-13 12:00:18,845 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testCorruptedSnapshot-1676289612809 2023-02-13 12:00:18,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 12:00:18,848 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289618848"}]},"ts":"1676289618848"} 2023-02-13 12:00:18,849 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=DISABLING in hbase:meta 2023-02-13 12:00:18,860 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676289612809 to state=DISABLING 2023-02-13 12:00:18,860 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN}] 2023-02-13 12:00:18,861 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN 2023-02-13 12:00:18,861 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN 2023-02-13 12:00:18,861 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN 2023-02-13 12:00:18,861 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN 2023-02-13 12:00:18,862 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN 2023-02-13 12:00:18,862 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:18,862 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289618862"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289618862"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289618862"}]},"ts":"1676289618862"} 2023-02-13 12:00:18,862 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:18,862 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:18,862 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289618862"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289618862"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289618862"}]},"ts":"1676289618862"} 2023-02-13 12:00:18,862 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:18,862 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289618862"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289618862"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289618862"}]},"ts":"1676289618862"} 2023-02-13 12:00:18,862 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:18,862 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289618862"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289618862"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289618862"}]},"ts":"1676289618862"} 2023-02-13 12:00:18,863 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289618862"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289618862"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289618862"}]},"ts":"1676289618862"} 2023-02-13 12:00:18,863 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=276, state=RUNNABLE; CloseRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:18,864 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=274, state=RUNNABLE; CloseRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:18,864 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=273, state=RUNNABLE; CloseRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518}] 2023-02-13 12:00:18,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=272, state=RUNNABLE; CloseRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:18,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=275, state=RUNNABLE; CloseRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427}] 2023-02-13 12:00:18,868 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN 2023-02-13 12:00:18,869 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:18,869 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289618869"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676289618869"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676289618869"}]},"ts":"1676289618869"} 2023-02-13 12:00:18,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478}] 2023-02-13 12:00:19,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:19,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:19,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing a0478ecfb8560aee2a65d010b27747ca, disabling compactions & flushes 2023-02-13 12:00:19,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:19,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:19,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. after waiting 0 ms 2023-02-13 12:00:19,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:19,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing f5abf54abb980cb8f138bde5b646e3c0, disabling compactions & flushes 2023-02-13 12:00:19,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing a0478ecfb8560aee2a65d010b27747ca 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-13 12:00:19,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:19,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:19,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. after waiting 0 ms 2023-02-13 12:00:19,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:19,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing f5abf54abb980cb8f138bde5b646e3c0 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-13 12:00:19,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:19,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 643a6d6a4f0efd58319e8e38b8b918fb, disabling compactions & flushes 2023-02-13 12:00:19,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:19,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:19,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. after waiting 0 ms 2023-02-13 12:00:19,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:19,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 643a6d6a4f0efd58319e8e38b8b918fb 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 12:00:19,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 12:00:19,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302132422c76ef33b47358d68bde2971aaf6e to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302132422c76ef33b47358d68bde2971aaf6e 2023-02-13 12:00:19,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021369d7a200ee3644d7bb8cd9583c652d96 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021369d7a200ee3644d7bb8cd9583c652d96 2023-02-13 12:00:19,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213b26ab9381ee44431acffd4af8ca36093 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213b26ab9381ee44431acffd4af8ca36093 2023-02-13 12:00:19,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/.tmp/cf/4c4c74ce40c64e5181adc5a3bf3a681c 2023-02-13 12:00:19,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/.tmp/cf/3251e9c0be9c4f5c8105b384647ebc46 2023-02-13 12:00:19,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/.tmp/cf/77bb12167e49493eaf982c875f4ea723 2023-02-13 12:00:19,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/.tmp/cf/4c4c74ce40c64e5181adc5a3bf3a681c as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/4c4c74ce40c64e5181adc5a3bf3a681c 2023-02-13 12:00:19,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/.tmp/cf/3251e9c0be9c4f5c8105b384647ebc46 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/3251e9c0be9c4f5c8105b384647ebc46 2023-02-13 12:00:19,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/.tmp/cf/77bb12167e49493eaf982c875f4ea723 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/77bb12167e49493eaf982c875f4ea723 2023-02-13 12:00:19,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/4c4c74ce40c64e5181adc5a3bf3a681c, entries=39, sequenceid=15, filesize=11.6 K 2023-02-13 12:00:19,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for a0478ecfb8560aee2a65d010b27747ca in 445ms, sequenceid=15, compaction requested=false 2023-02-13 12:00:19,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/3251e9c0be9c4f5c8105b384647ebc46, entries=31, sequenceid=15, filesize=10.2 K 2023-02-13 12:00:19,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 643a6d6a4f0efd58319e8e38b8b918fb in 443ms, sequenceid=15, compaction requested=false 2023-02-13 12:00:19,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/77bb12167e49493eaf982c875f4ea723, entries=35, sequenceid=15, filesize=10.9 K 2023-02-13 12:00:19,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for f5abf54abb980cb8f138bde5b646e3c0 in 448ms, sequenceid=15, compaction requested=false 2023-02-13 12:00:19,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:00:19,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. 2023-02-13 12:00:19,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for a0478ecfb8560aee2a65d010b27747ca: Waiting for close lock at 1676289619016Running coprocessor pre-close hooks at 1676289619016Disabling compacts and flushes for region at 1676289619016Disabling writes for close at 1676289619017 (+1 ms)Obtaining lock to block concurrent updates at 1676289619018 (+1 ms)Preparing flush snapshotting stores in a0478ecfb8560aee2a65d010b27747ca at 1676289619018Finished memstore snapshotting testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676289619018Flushing stores of testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca. at 1676289619019 (+1 ms)Flushing a0478ecfb8560aee2a65d010b27747ca/cf: creating writer at 1676289619022 (+3 ms)Flushing mob file a0478ecfb8560aee2a65d010b27747ca/cf: appending metadata at 1676289619028 (+6 ms)Flushing mob file a0478ecfb8560aee2a65d010b27747ca/cf: closing flushed file at 1676289619028Flushing a0478ecfb8560aee2a65d010b27747ca/cf: appending metadata at 1676289619446 (+418 ms)Flushing a0478ecfb8560aee2a65d010b27747ca/cf: closing flushed file at 1676289619446Flushing a0478ecfb8560aee2a65d010b27747ca/cf: reopening flushed file at 1676289619459 (+13 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for a0478ecfb8560aee2a65d010b27747ca in 445ms, sequenceid=15, compaction requested=false at 1676289619462 (+3 ms)Writing region close event to WAL at 1676289619462Running coprocessor post-close hooks at 1676289619467 (+5 ms)Closed at 1676289619468 (+1 ms) 2023-02-13 12:00:19,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:00:19,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. 2023-02-13 12:00:19,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 643a6d6a4f0efd58319e8e38b8b918fb: Waiting for close lock at 1676289619022Running coprocessor pre-close hooks at 1676289619022Disabling compacts and flushes for region at 1676289619022Disabling writes for close at 1676289619022Obtaining lock to block concurrent updates at 1676289619023 (+1 ms)Preparing flush snapshotting stores in 643a6d6a4f0efd58319e8e38b8b918fb at 1676289619023Finished memstore snapshotting testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676289619024 (+1 ms)Flushing stores of testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb. at 1676289619025 (+1 ms)Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: creating writer at 1676289619026 (+1 ms)Flushing mob file 643a6d6a4f0efd58319e8e38b8b918fb/cf: appending metadata at 1676289619029 (+3 ms)Flushing mob file 643a6d6a4f0efd58319e8e38b8b918fb/cf: closing flushed file at 1676289619029Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: appending metadata at 1676289619445 (+416 ms)Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: closing flushed file at 1676289619446 (+1 ms)Flushing 643a6d6a4f0efd58319e8e38b8b918fb/cf: reopening flushed file at 1676289619460 (+14 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 643a6d6a4f0efd58319e8e38b8b918fb in 443ms, sequenceid=15, compaction requested=false at 1676289619465 (+5 ms)Writing region close event to WAL at 1676289619465Running coprocessor post-close hooks at 1676289619469 (+4 ms)Closed at 1676289619470 (+1 ms) 2023-02-13 12:00:19,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:19,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:19,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 48d6107ef6392fc0224ecd69dad7d2f3, disabling compactions & flushes 2023-02-13 12:00:19,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:19,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:19,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. after waiting 0 ms 2023-02-13 12:00:19,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:00:19,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:19,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 48d6107ef6392fc0224ecd69dad7d2f3 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-13 12:00:19,472 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. 2023-02-13 12:00:19,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for f5abf54abb980cb8f138bde5b646e3c0: Waiting for close lock at 1676289619017Running coprocessor pre-close hooks at 1676289619017Disabling compacts and flushes for region at 1676289619017Disabling writes for close at 1676289619018 (+1 ms)Obtaining lock to block concurrent updates at 1676289619018Preparing flush snapshotting stores in f5abf54abb980cb8f138bde5b646e3c0 at 1676289619018Finished memstore snapshotting testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0., syncing WAL and waiting on mvcc, flushsize=dataSize=2343, getHeapSize=5280, getOffHeapSize=0, getCellsCount=35 at 1676289619019 (+1 ms)Flushing stores of testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0. at 1676289619023 (+4 ms)Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: creating writer at 1676289619024 (+1 ms)Flushing mob file f5abf54abb980cb8f138bde5b646e3c0/cf: appending metadata at 1676289619029 (+5 ms)Flushing mob file f5abf54abb980cb8f138bde5b646e3c0/cf: closing flushed file at 1676289619029Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: appending metadata at 1676289619446 (+417 ms)Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: closing flushed file at 1676289619446Flushing f5abf54abb980cb8f138bde5b646e3c0/cf: reopening flushed file at 1676289619461 (+15 ms)Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for f5abf54abb980cb8f138bde5b646e3c0 in 448ms, sequenceid=15, compaction requested=false at 1676289619466 (+5 ms)Writing region close event to WAL at 1676289619466Running coprocessor post-close hooks at 1676289619471 (+5 ms)Closed at 1676289619472 (+1 ms) 2023-02-13 12:00:19,472 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=a0478ecfb8560aee2a65d010b27747ca, regionState=CLOSED 2023-02-13 12:00:19,472 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289619472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289619472"}]},"ts":"1676289619472"} 2023-02-13 12:00:19,472 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:19,472 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:19,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 9bdb004dd508672ca8f98b4ad3d79f3a, disabling compactions & flushes 2023-02-13 12:00:19,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:19,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:19,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. after waiting 0 ms 2023-02-13 12:00:19,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:19,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 9bdb004dd508672ca8f98b4ad3d79f3a 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-13 12:00:19,473 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=643a6d6a4f0efd58319e8e38b8b918fb, regionState=CLOSED 2023-02-13 12:00:19,474 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289619473"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289619473"}]},"ts":"1676289619473"} 2023-02-13 12:00:19,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:19,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:19,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a6a2c6c5a450fab5269836430582d6, disabling compactions & flushes 2023-02-13 12:00:19,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:19,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:19,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. after waiting 0 ms 2023-02-13 12:00:19,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:19,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 81a6a2c6c5a450fab5269836430582d6 1/1 column families, dataSize=21.46 KB heapSize=46.38 KB 2023-02-13 12:00:19,475 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=f5abf54abb980cb8f138bde5b646e3c0, regionState=CLOSED 2023-02-13 12:00:19,475 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289619475"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289619475"}]},"ts":"1676289619475"} 2023-02-13 12:00:19,478 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=274 2023-02-13 12:00:19,478 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=274, state=SUCCESS; CloseRegionProcedure a0478ecfb8560aee2a65d010b27747ca, server=jenkins-hbase16.apache.org,34233,1676289577518 in 611 msec 2023-02-13 12:00:19,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=275 2023-02-13 12:00:19,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=275, state=SUCCESS; CloseRegionProcedure 643a6d6a4f0efd58319e8e38b8b918fb, server=jenkins-hbase16.apache.org,35305,1676289577427 in 612 msec 2023-02-13 12:00:19,479 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-13 12:00:19,479 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=a0478ecfb8560aee2a65d010b27747ca, UNASSIGN in 618 msec 2023-02-13 12:00:19,479 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure f5abf54abb980cb8f138bde5b646e3c0, server=jenkins-hbase16.apache.org,45135,1676289577478 in 607 msec 2023-02-13 12:00:19,481 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=643a6d6a4f0efd58319e8e38b8b918fb, UNASSIGN in 619 msec 2023-02-13 12:00:19,481 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=f5abf54abb980cb8f138bde5b646e3c0, UNASSIGN in 619 msec 2023-02-13 12:00:19,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213504f7b39f691454a83301212823a90d9 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b20230213504f7b39f691454a83301212823a90d9 2023-02-13 12:00:19,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213a8d6e62e734341a8a491462033c83dc1 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c20230213a8d6e62e734341a8a491462033c83dc1 2023-02-13 12:00:19,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230213138109b720f24a69ae429b595f1412e9 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d520230213138109b720f24a69ae429b595f1412e9 2023-02-13 12:00:19,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/.tmp/cf/ff79cc23aa414304a8e886ab5c435801 2023-02-13 12:00:19,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/.tmp/cf/4228f676b2274676aa74f1eb143f5351 2023-02-13 12:00:19,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/.tmp/cf/ff79cc23aa414304a8e886ab5c435801 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/ff79cc23aa414304a8e886ab5c435801 2023-02-13 12:00:19,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/.tmp/cf/4228f676b2274676aa74f1eb143f5351 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/4228f676b2274676aa74f1eb143f5351 2023-02-13 12:00:19,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/.tmp/cf/938a2d4ac269459688649439697a00e9 2023-02-13 12:00:19,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/4228f676b2274676aa74f1eb143f5351, entries=29, sequenceid=15, filesize=9.9 K 2023-02-13 12:00:19,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/ff79cc23aa414304a8e886ab5c435801, entries=38, sequenceid=15, filesize=11.4 K 2023-02-13 12:00:19,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/.tmp/cf/938a2d4ac269459688649439697a00e9 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/938a2d4ac269459688649439697a00e9 2023-02-13 12:00:19,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 48d6107ef6392fc0224ecd69dad7d2f3 in 34ms, sequenceid=15, compaction requested=false 2023-02-13 12:00:19,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 9bdb004dd508672ca8f98b4ad3d79f3a in 32ms, sequenceid=15, compaction requested=false 2023-02-13 12:00:19,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/938a2d4ac269459688649439697a00e9, entries=328, sequenceid=15, filesize=61.4 K 2023-02-13 12:00:19,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:00:19,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:00:19,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. 2023-02-13 12:00:19,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. 2023-02-13 12:00:19,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 48d6107ef6392fc0224ecd69dad7d2f3: Waiting for close lock at 1676289619470Running coprocessor pre-close hooks at 1676289619470Disabling compacts and flushes for region at 1676289619470Disabling writes for close at 1676289619470Obtaining lock to block concurrent updates at 1676289619471 (+1 ms)Preparing flush snapshotting stores in 48d6107ef6392fc0224ecd69dad7d2f3 at 1676289619471Finished memstore snapshotting testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676289619471Flushing stores of testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3. at 1676289619472 (+1 ms)Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: creating writer at 1676289619473 (+1 ms)Flushing mob file 48d6107ef6392fc0224ecd69dad7d2f3/cf: appending metadata at 1676289619480 (+7 ms)Flushing mob file 48d6107ef6392fc0224ecd69dad7d2f3/cf: closing flushed file at 1676289619480Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: appending metadata at 1676289619492 (+12 ms)Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: closing flushed file at 1676289619492Flushing 48d6107ef6392fc0224ecd69dad7d2f3/cf: reopening flushed file at 1676289619502 (+10 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 48d6107ef6392fc0224ecd69dad7d2f3 in 34ms, sequenceid=15, compaction requested=false at 1676289619505 (+3 ms)Writing region close event to WAL at 1676289619506 (+1 ms)Running coprocessor post-close hooks at 1676289619509 (+3 ms)Closed at 1676289619510 (+1 ms) 2023-02-13 12:00:19,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 9bdb004dd508672ca8f98b4ad3d79f3a: Waiting for close lock at 1676289619473Running coprocessor pre-close hooks at 1676289619473Disabling compacts and flushes for region at 1676289619473Disabling writes for close at 1676289619473Obtaining lock to block concurrent updates at 1676289619474 (+1 ms)Preparing flush snapshotting stores in 9bdb004dd508672ca8f98b4ad3d79f3a at 1676289619474Finished memstore snapshotting testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676289619474Flushing stores of testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a. at 1676289619474Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: creating writer at 1676289619475 (+1 ms)Flushing mob file 9bdb004dd508672ca8f98b4ad3d79f3a/cf: appending metadata at 1676289619481 (+6 ms)Flushing mob file 9bdb004dd508672ca8f98b4ad3d79f3a/cf: closing flushed file at 1676289619481Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: appending metadata at 1676289619492 (+11 ms)Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: closing flushed file at 1676289619492Flushing 9bdb004dd508672ca8f98b4ad3d79f3a/cf: reopening flushed file at 1676289619502 (+10 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 9bdb004dd508672ca8f98b4ad3d79f3a in 32ms, sequenceid=15, compaction requested=false at 1676289619505 (+3 ms)Writing region close event to WAL at 1676289619506 (+1 ms)Running coprocessor post-close hooks at 1676289619509 (+3 ms)Closed at 1676289619510 (+1 ms) 2023-02-13 12:00:19,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.46 KB/21972, heapSize ~46.36 KB/47472, currentSize=0 B/0 for 81a6a2c6c5a450fab5269836430582d6 in 36ms, sequenceid=15, compaction requested=false 2023-02-13 12:00:19,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:19,511 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=48d6107ef6392fc0224ecd69dad7d2f3, regionState=CLOSED 2023-02-13 12:00:19,511 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289619511"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289619511"}]},"ts":"1676289619511"} 2023-02-13 12:00:19,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:19,512 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=9bdb004dd508672ca8f98b4ad3d79f3a, regionState=CLOSED 2023-02-13 12:00:19,512 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676289619512"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289619512"}]},"ts":"1676289619512"} 2023-02-13 12:00:19,512 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:00:19,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. 2023-02-13 12:00:19,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a6a2c6c5a450fab5269836430582d6: Waiting for close lock at 1676289619474Running coprocessor pre-close hooks at 1676289619474Disabling compacts and flushes for region at 1676289619474Disabling writes for close at 1676289619474Obtaining lock to block concurrent updates at 1676289619474Preparing flush snapshotting stores in 81a6a2c6c5a450fab5269836430582d6 at 1676289619474Finished memstore snapshotting testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6., syncing WAL and waiting on mvcc, flushsize=dataSize=21972, getHeapSize=47472, getOffHeapSize=0, getCellsCount=328 at 1676289619474Flushing stores of testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6. at 1676289619475 (+1 ms)Flushing 81a6a2c6c5a450fab5269836430582d6/cf: creating writer at 1676289619477 (+2 ms)Flushing mob file 81a6a2c6c5a450fab5269836430582d6/cf: appending metadata at 1676289619484 (+7 ms)Flushing mob file 81a6a2c6c5a450fab5269836430582d6/cf: closing flushed file at 1676289619484Flushing 81a6a2c6c5a450fab5269836430582d6/cf: appending metadata at 1676289619495 (+11 ms)Flushing 81a6a2c6c5a450fab5269836430582d6/cf: closing flushed file at 1676289619495Flushing 81a6a2c6c5a450fab5269836430582d6/cf: reopening flushed file at 1676289619506 (+11 ms)Finished flush of dataSize ~21.46 KB/21972, heapSize ~46.36 KB/47472, currentSize=0 B/0 for 81a6a2c6c5a450fab5269836430582d6 in 36ms, sequenceid=15, compaction requested=false at 1676289619510 (+4 ms)Writing region close event to WAL at 1676289619510Running coprocessor post-close hooks at 1676289619513 (+3 ms)Closed at 1676289619514 (+1 ms) 2023-02-13 12:00:19,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=273 2023-02-13 12:00:19,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=273, state=SUCCESS; CloseRegionProcedure 48d6107ef6392fc0224ecd69dad7d2f3, server=jenkins-hbase16.apache.org,34233,1676289577518 in 648 msec 2023-02-13 12:00:19,515 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=272 2023-02-13 12:00:19,515 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=272, state=SUCCESS; CloseRegionProcedure 9bdb004dd508672ca8f98b4ad3d79f3a, server=jenkins-hbase16.apache.org,35305,1676289577427 in 648 msec 2023-02-13 12:00:19,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:19,515 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=81a6a2c6c5a450fab5269836430582d6, regionState=CLOSED 2023-02-13 12:00:19,515 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=48d6107ef6392fc0224ecd69dad7d2f3, UNASSIGN in 654 msec 2023-02-13 12:00:19,515 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676289619515"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676289619515"}]},"ts":"1676289619515"} 2023-02-13 12:00:19,515 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=9bdb004dd508672ca8f98b4ad3d79f3a, UNASSIGN in 655 msec 2023-02-13 12:00:19,517 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=276 2023-02-13 12:00:19,517 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=276, state=SUCCESS; CloseRegionProcedure 81a6a2c6c5a450fab5269836430582d6, server=jenkins-hbase16.apache.org,45135,1676289577478 in 653 msec 2023-02-13 12:00:19,517 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=276, resume processing ppid=270 2023-02-13 12:00:19,517 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676289612809, region=81a6a2c6c5a450fab5269836430582d6, UNASSIGN in 657 msec 2023-02-13 12:00:19,518 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676289619518"}]},"ts":"1676289619518"} 2023-02-13 12:00:19,519 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676289612809, state=DISABLED in hbase:meta 2023-02-13 12:00:19,526 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676289612809 to state=DISABLED 2023-02-13 12:00:19,528 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676289612809 in 682 msec 2023-02-13 12:00:19,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 12:00:19,601 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676289612809, procId: 270 completed 2023-02-13 12:00:19,602 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.HMaster$5(2215): Client=jenkins//188.40.62.62 delete testCorruptedSnapshot-1676289612809 2023-02-13 12:00:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:19,604 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:19,605 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 12:00:19,612 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:19,612 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:19,612 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:19,612 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:19,612 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:19,612 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:19,614 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/recovered.edits] 2023-02-13 12:00:19,614 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/recovered.edits] 2023-02-13 12:00:19,615 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/recovered.edits] 2023-02-13 12:00:19,615 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/recovered.edits] 2023-02-13 12:00:19,615 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/recovered.edits] 2023-02-13 12:00:19,615 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf, FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/recovered.edits] 2023-02-13 12:00:19,618 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/53ceb530421c4eb1b807a459c580ec75 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/53ceb530421c4eb1b807a459c580ec75 2023-02-13 12:00:19,619 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/f989a8e47ab847c8a8e2f4de64587c29 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/f989a8e47ab847c8a8e2f4de64587c29 2023-02-13 12:00:19,619 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/4228f676b2274676aa74f1eb143f5351 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/cf/4228f676b2274676aa74f1eb143f5351 2023-02-13 12:00:19,619 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/2ce8d42bb8dc4433ab86bdffdc44b465 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/2ce8d42bb8dc4433ab86bdffdc44b465 2023-02-13 12:00:19,620 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/938a2d4ac269459688649439697a00e9 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/938a2d4ac269459688649439697a00e9 2023-02-13 12:00:19,620 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/77bb12167e49493eaf982c875f4ea723 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/cf/77bb12167e49493eaf982c875f4ea723 2023-02-13 12:00:19,620 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/4c4c74ce40c64e5181adc5a3bf3a681c to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/4c4c74ce40c64e5181adc5a3bf3a681c 2023-02-13 12:00:19,621 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/ff79cc23aa414304a8e886ab5c435801 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/cf/ff79cc23aa414304a8e886ab5c435801 2023-02-13 12:00:19,621 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/3251e9c0be9c4f5c8105b384647ebc46 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/cf/3251e9c0be9c4f5c8105b384647ebc46 2023-02-13 12:00:19,621 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/980a841c48a64c50ac0f46aa4f34b1fb to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/cf/980a841c48a64c50ac0f46aa4f34b1fb 2023-02-13 12:00:19,622 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/c2a7ca3b619048038c36f07886ad7fa7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/cf/c2a7ca3b619048038c36f07886ad7fa7 2023-02-13 12:00:19,622 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3/recovered.edits/18.seqid 2023-02-13 12:00:19,623 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/48d6107ef6392fc0224ecd69dad7d2f3 2023-02-13 12:00:19,623 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0/recovered.edits/18.seqid 2023-02-13 12:00:19,624 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/f5abf54abb980cb8f138bde5b646e3c0 2023-02-13 12:00:19,624 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a/recovered.edits/18.seqid 2023-02-13 12:00:19,624 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6/recovered.edits/18.seqid 2023-02-13 12:00:19,625 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb/recovered.edits/18.seqid 2023-02-13 12:00:19,625 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/9bdb004dd508672ca8f98b4ad3d79f3a 2023-02-13 12:00:19,625 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/81a6a2c6c5a450fab5269836430582d6 2023-02-13 12:00:19,625 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/643a6d6a4f0efd58319e8e38b8b918fb 2023-02-13 12:00:19,625 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/recovered.edits/18.seqid to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca/recovered.edits/18.seqid 2023-02-13 12:00:19,626 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/.tmp/data/default/testCorruptedSnapshot-1676289612809/a0478ecfb8560aee2a65d010b27747ca 2023-02-13 12:00:19,626 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676289612809 regions 2023-02-13 12:00:19,626 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47 2023-02-13 12:00:19,627 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf] 2023-02-13 12:00:19,629 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302131dd1c12e26a340a6acf0fc609d79dace to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302131dd1c12e26a340a6acf0fc609d79dace 2023-02-13 12:00:19,630 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302132422c76ef33b47358d68bde2971aaf6e to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/a87ff679a2f3e71d9181a67b7542122c202302132422c76ef33b47358d68bde2971aaf6e 2023-02-13 12:00:19,631 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b20230213504f7b39f691454a83301212823a90d9 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b20230213504f7b39f691454a83301212823a90d9 2023-02-13 12:00:19,632 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b202302138d0ecd628abb4f82ac488c033938186b to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c4ca4238a0b923820dcc509a6f75849b202302138d0ecd628abb4f82ac488c033938186b 2023-02-13 12:00:19,633 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c2023021357f9f893317244efb84943eb9cb531b7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c2023021357f9f893317244efb84943eb9cb531b7 2023-02-13 12:00:19,634 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c20230213a8d6e62e734341a8a491462033c83dc1 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/c81e728d9d4c2f636f067f89cc14862c20230213a8d6e62e734341a8a491462033c83dc1 2023-02-13 12:00:19,635 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213829051f01ee2458ab84a8d40fba682a7 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213829051f01ee2458ab84a8d40fba682a7 2023-02-13 12:00:19,636 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213b26ab9381ee44431acffd4af8ca36093 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/d41d8cd98f00b204e9800998ecf8427e20230213b26ab9381ee44431acffd4af8ca36093 2023-02-13 12:00:19,637 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d520230213138109b720f24a69ae429b595f1412e9 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d520230213138109b720f24a69ae429b595f1412e9 2023-02-13 12:00:19,638 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d52023021348d3eb972f194efdbeddfeaaf53481ba to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/e4da3b7fbbce2345d7772b0674a318d52023021348d3eb972f194efdbeddfeaaf53481ba 2023-02-13 12:00:19,639 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021369d7a200ee3644d7bb8cd9583c652d96 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021369d7a200ee3644d7bb8cd9583c652d96 2023-02-13 12:00:19,641 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d9f7e3c6e703460ea1c207b0f8651869 to hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/archive/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213d9f7e3c6e703460ea1c207b0f8651869 2023-02-13 12:00:19,641 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/mobdir/data/default/testCorruptedSnapshot-1676289612809/fd902ff2013a67ee277736586155df47 2023-02-13 12:00:19,643 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:19,645 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676289612809 from hbase:meta 2023-02-13 12:00:19,647 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676289612809' descriptor. 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676289612809' from region states. 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289619648"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289619648"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289619648"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289619648"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289619648"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:19,648 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676289619648"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:19,650 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:00:19,650 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f5abf54abb980cb8f138bde5b646e3c0, NAME => 'testCorruptedSnapshot-1676289612809,,1676289612811.f5abf54abb980cb8f138bde5b646e3c0.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 9bdb004dd508672ca8f98b4ad3d79f3a, NAME => 'testCorruptedSnapshot-1676289612809,1,1676289612811.9bdb004dd508672ca8f98b4ad3d79f3a.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 48d6107ef6392fc0224ecd69dad7d2f3, NAME => 'testCorruptedSnapshot-1676289612809,2,1676289612811.48d6107ef6392fc0224ecd69dad7d2f3.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => a0478ecfb8560aee2a65d010b27747ca, NAME => 'testCorruptedSnapshot-1676289612809,3,1676289612811.a0478ecfb8560aee2a65d010b27747ca.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 643a6d6a4f0efd58319e8e38b8b918fb, NAME => 'testCorruptedSnapshot-1676289612809,4,1676289612811.643a6d6a4f0efd58319e8e38b8b918fb.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 81a6a2c6c5a450fab5269836430582d6, NAME => 'testCorruptedSnapshot-1676289612809,5,1676289612811.81a6a2c6c5a450fab5269836430582d6.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 12:00:19,650 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676289612809' as deleted. 2023-02-13 12:00:19,650 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676289612809","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676289619650"}]},"ts":"9223372036854775807"} 2023-02-13 12:00:19,651 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676289612809 state from META 2023-02-13 12:00:19,660 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676289612809 2023-02-13 12:00:19,661 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676289612809 in 58 msec 2023-02-13 12:00:19,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 12:00:19,857 INFO [Listener at localhost.localdomain/36409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676289612809, procId: 283 completed 2023-02-13 12:00:19,871 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "emptySnaptb-1676289612809" 2023-02-13 12:00:19,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676289612809 2023-02-13 12:00:19,875 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "snaptb0-1676289612809" 2023-02-13 12:00:19,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39205] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676289612809 2023-02-13 12:00:19,901 INFO [Listener at localhost.localdomain/36409] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=424 (was 409) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:52212 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2075941959_15 at /127.0.0.1:59838 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1826247995_15 at /127.0.0.1:52120 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6136c840-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:52936 [Waiting for operation #125] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6136c840-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:52148 [Waiting for operation #266] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1710913709_15 at /127.0.0.1:49542 [Waiting for operation #506] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-120966320_15 at /127.0.0.1:52132 [Waiting for operation #288] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2075941959_15 at /127.0.0.1:52920 [Waiting for operation #129] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1710913709_15 at /127.0.0.1:52910 [Waiting for operation #139] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-541515498_15 at /127.0.0.1:58490 [Waiting for operation #521] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6136c840-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1b9902f2-shared-pool-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1812496918_15 at /127.0.0.1:58474 [Waiting for operation #545] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6136c840-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2021159335_15 at /127.0.0.1:60196 [Waiting for operation #269] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1826247995_15 at /127.0.0.1:51896 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-541515498_15 at /127.0.0.1:52782 [Waiting for operation #132] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6136c840-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1826247995_15 at /127.0.0.1:49556 [Waiting for operation #757] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=798 (was 793) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=377 (was 446), ProcessCount=169 (was 171), AvailableMemoryMB=4461 (was 4515) 2023-02-13 12:00:19,902 INFO [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 12:00:19,902 INFO [Listener at localhost.localdomain/36409] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 12:00:19,902 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76012f12 to 127.0.0.1:63172 2023-02-13 12:00:19,903 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:19,903 DEBUG [Listener at localhost.localdomain/36409] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 12:00:19,903 DEBUG [Listener at localhost.localdomain/36409] util.JVMClusterUtil(257): Found active master hash=523612691, stopped=false 2023-02-13 12:00:19,903 INFO [Listener at localhost.localdomain/36409] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 12:00:19,919 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:00:19,919 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:00:19,919 INFO [Listener at localhost.localdomain/36409] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 12:00:19,919 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:00:19,919 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:00:19,920 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:00:19,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:00:19,920 DEBUG [Listener at localhost.localdomain/36409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x140f10ec to 127.0.0.1:63172 2023-02-13 12:00:19,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:00:19,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:00:19,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:00:19,920 DEBUG [Listener at localhost.localdomain/36409] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:19,921 INFO [Listener at localhost.localdomain/36409] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,35305,1676289577427' ***** 2023-02-13 12:00:19,921 INFO [Listener at localhost.localdomain/36409] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 12:00:19,921 INFO [Listener at localhost.localdomain/36409] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,45135,1676289577478' ***** 2023-02-13 12:00:19,921 INFO [Listener at localhost.localdomain/36409] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 12:00:19,921 INFO [Listener at localhost.localdomain/36409] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,34233,1676289577518' ***** 2023-02-13 12:00:19,921 INFO [Listener at localhost.localdomain/36409] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 12:00:19,921 INFO [RS:0;jenkins-hbase16:35305] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 12:00:19,921 INFO [RS:2;jenkins-hbase16:34233] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 12:00:19,921 INFO [RS:1;jenkins-hbase16:45135] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 12:00:19,921 INFO [RS:0;jenkins-hbase16:35305] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 12:00:19,921 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 12:00:19,922 INFO [RS:0;jenkins-hbase16:35305] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 12:00:19,921 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 12:00:19,921 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 12:00:19,921 INFO [RS:1;jenkins-hbase16:45135] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 12:00:19,921 INFO [RS:2;jenkins-hbase16:34233] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 12:00:19,922 INFO [RS:1;jenkins-hbase16:45135] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 12:00:19,922 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:19,922 INFO [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:19,922 DEBUG [RS:0;jenkins-hbase16:35305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aa55d4b to 127.0.0.1:63172 2023-02-13 12:00:19,922 INFO [RS:2;jenkins-hbase16:34233] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 12:00:19,922 DEBUG [RS:0;jenkins-hbase16:35305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:19,922 DEBUG [RS:1;jenkins-hbase16:45135] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ae8888a to 127.0.0.1:63172 2023-02-13 12:00:19,922 INFO [RS:0;jenkins-hbase16:35305] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 12:00:19,922 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(3304): Received CLOSE for df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 12:00:19,923 INFO [RS:0;jenkins-hbase16:35305] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 12:00:19,922 DEBUG [RS:1;jenkins-hbase16:45135] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:19,923 INFO [RS:0;jenkins-hbase16:35305] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 12:00:19,923 INFO [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,45135,1676289577478; all regions closed. 2023-02-13 12:00:19,923 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 12:00:19,923 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:19,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing df3c86cd8ebebefe70d222a681f7de3f, disabling compactions & flushes 2023-02-13 12:00:19,923 DEBUG [RS:2;jenkins-hbase16:34233] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46789225 to 127.0.0.1:63172 2023-02-13 12:00:19,923 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 12:00:19,923 DEBUG [RS:2;jenkins-hbase16:34233] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:19,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 12:00:19,923 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 12:00:19,923 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-13 12:00:19,923 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 12:00:19,923 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1479): Online Regions={df3c86cd8ebebefe70d222a681f7de3f=hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f.} 2023-02-13 12:00:19,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 12:00:19,924 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 12:00:19,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. after waiting 0 ms 2023-02-13 12:00:19,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 12:00:19,924 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 12:00:19,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing df3c86cd8ebebefe70d222a681f7de3f 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 12:00:19,924 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 12:00:19,924 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 12:00:19,924 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.88 KB heapSize=212.75 KB 2023-02-13 12:00:19,926 DEBUG [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1505): Waiting on df3c86cd8ebebefe70d222a681f7de3f 2023-02-13 12:00:19,926 DEBUG [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-13 12:00:19,932 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 12:00:19,933 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 12:00:19,935 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:00:19,935 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:00:19,935 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:00:19,941 DEBUG [RS:1;jenkins-hbase16:45135] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs 2023-02-13 12:00:19,941 INFO [RS:1;jenkins-hbase16:45135] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C45135%2C1676289577478:(num 1676289579117) 2023-02-13 12:00:19,941 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 12:00:19,941 DEBUG [RS:1;jenkins-hbase16:45135] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:19,941 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 12:00:19,941 INFO [RS:1;jenkins-hbase16:45135] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:00:19,951 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 12:00:19,951 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 12:00:19,952 INFO [RS:1;jenkins-hbase16:45135] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 12:00:19,952 INFO [RS:1;jenkins-hbase16:45135] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 12:00:19,952 INFO [RS:1;jenkins-hbase16:45135] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 12:00:19,952 INFO [RS:1;jenkins-hbase16:45135] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 12:00:19,952 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:00:19,954 INFO [RS:1;jenkins-hbase16:45135] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:45135 2023-02-13 12:00:19,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/.tmp/info/f6ef2a1e31be4afea7a1971d295db394 2023-02-13 12:00:19,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/.tmp/info/f6ef2a1e31be4afea7a1971d295db394 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/info/f6ef2a1e31be4afea7a1971d295db394 2023-02-13 12:00:19,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/info/f6ef2a1e31be4afea7a1971d295db394, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 12:00:19,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for df3c86cd8ebebefe70d222a681f7de3f in 57ms, sequenceid=6, compaction requested=false 2023-02-13 12:00:19,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/namespace/df3c86cd8ebebefe70d222a681f7de3f/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 12:00:19,997 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=131.12 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/.tmp/info/bbace76013f04955ad5057c5e84a7398 2023-02-13 12:00:19,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 12:00:19,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for df3c86cd8ebebefe70d222a681f7de3f: Waiting for close lock at 1676289619923Running coprocessor pre-close hooks at 1676289619923Disabling compacts and flushes for region at 1676289619923Disabling writes for close at 1676289619924 (+1 ms)Obtaining lock to block concurrent updates at 1676289619924Preparing flush snapshotting stores in df3c86cd8ebebefe70d222a681f7de3f at 1676289619924Finished memstore snapshotting hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676289619924Flushing stores of hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. at 1676289619925 (+1 ms)Flushing df3c86cd8ebebefe70d222a681f7de3f/info: creating writer at 1676289619926 (+1 ms)Flushing df3c86cd8ebebefe70d222a681f7de3f/info: appending metadata at 1676289619940 (+14 ms)Flushing df3c86cd8ebebefe70d222a681f7de3f/info: closing flushed file at 1676289619940Flushing df3c86cd8ebebefe70d222a681f7de3f/info: reopening flushed file at 1676289619977 (+37 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for df3c86cd8ebebefe70d222a681f7de3f in 57ms, sequenceid=6, compaction requested=false at 1676289619981 (+4 ms)Writing region close event to WAL at 1676289619990 (+9 ms)Running coprocessor post-close hooks at 1676289619997 (+7 ms)Closed at 1676289619998 (+1 ms) 2023-02-13 12:00:19,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676289579490.df3c86cd8ebebefe70d222a681f7de3f. 2023-02-13 12:00:20,002 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bbace76013f04955ad5057c5e84a7398 2023-02-13 12:00:20,029 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/.tmp/rep_barrier/a324ac0f77244de09d70fc1d2f305a99 2023-02-13 12:00:20,032 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a324ac0f77244de09d70fc1d2f305a99 2023-02-13 12:00:20,041 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:20,041 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:20,041 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:00:20,041 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,45135,1676289577478 2023-02-13 12:00:20,041 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:00:20,041 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:00:20,041 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:00:20,044 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/.tmp/table/f2b3fe0484fc47b694bed39b50573b59 2023-02-13 12:00:20,046 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f2b3fe0484fc47b694bed39b50573b59 2023-02-13 12:00:20,047 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/.tmp/info/bbace76013f04955ad5057c5e84a7398 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/info/bbace76013f04955ad5057c5e84a7398 2023-02-13 12:00:20,049 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bbace76013f04955ad5057c5e84a7398 2023-02-13 12:00:20,049 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/info/bbace76013f04955ad5057c5e84a7398, entries=64, sequenceid=307, filesize=11.9 K 2023-02-13 12:00:20,050 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/.tmp/rep_barrier/a324ac0f77244de09d70fc1d2f305a99 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/rep_barrier/a324ac0f77244de09d70fc1d2f305a99 2023-02-13 12:00:20,053 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a324ac0f77244de09d70fc1d2f305a99 2023-02-13 12:00:20,053 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/rep_barrier/a324ac0f77244de09d70fc1d2f305a99, entries=18, sequenceid=307, filesize=7.0 K 2023-02-13 12:00:20,053 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/.tmp/table/f2b3fe0484fc47b694bed39b50573b59 as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/table/f2b3fe0484fc47b694bed39b50573b59 2023-02-13 12:00:20,056 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f2b3fe0484fc47b694bed39b50573b59 2023-02-13 12:00:20,056 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/table/f2b3fe0484fc47b694bed39b50573b59, entries=22, sequenceid=307, filesize=7.0 K 2023-02-13 12:00:20,057 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 133ms, sequenceid=307, compaction requested=false 2023-02-13 12:00:20,062 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-13 12:00:20,063 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 12:00:20,064 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 12:00:20,064 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676289619923Running coprocessor pre-close hooks at 1676289619923Disabling compacts and flushes for region at 1676289619923Disabling writes for close at 1676289619924 (+1 ms)Obtaining lock to block concurrent updates at 1676289619924Preparing flush snapshotting stores in 1588230740 at 1676289619924Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141192, getHeapSize=217808, getOffHeapSize=0, getCellsCount=997 at 1676289619925 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676289619926 (+1 ms)Flushing 1588230740/info: creating writer at 1676289619926Flushing 1588230740/info: appending metadata at 1676289619932 (+6 ms)Flushing 1588230740/info: closing flushed file at 1676289619932Flushing 1588230740/rep_barrier: creating writer at 1676289620002 (+70 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676289620008 (+6 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676289620008Flushing 1588230740/table: creating writer at 1676289620032 (+24 ms)Flushing 1588230740/table: appending metadata at 1676289620034 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676289620035 (+1 ms)Flushing 1588230740/info: reopening flushed file at 1676289620047 (+12 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676289620050 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676289620054 (+4 ms)Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 133ms, sequenceid=307, compaction requested=false at 1676289620057 (+3 ms)Writing region close event to WAL at 1676289620059 (+2 ms)Running coprocessor post-close hooks at 1676289620063 (+4 ms)Closed at 1676289620064 (+1 ms) 2023-02-13 12:00:20,064 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 12:00:20,126 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,34233,1676289577518; all regions closed. 2023-02-13 12:00:20,126 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,35305,1676289577427; all regions closed. 2023-02-13 12:00:20,136 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,45135,1676289577478] 2023-02-13 12:00:20,136 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,45135,1676289577478; numProcessing=1 2023-02-13 12:00:20,138 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/WALs/jenkins-hbase16.apache.org,34233,1676289577518/jenkins-hbase16.apache.org%2C34233%2C1676289577518.1676289579117 not finished, retry = 0 2023-02-13 12:00:20,139 DEBUG [RS:0;jenkins-hbase16:35305] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs 2023-02-13 12:00:20,139 INFO [RS:0;jenkins-hbase16:35305] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C35305%2C1676289577427.meta:.meta(num 1676289579210) 2023-02-13 12:00:20,145 DEBUG [RS:0;jenkins-hbase16:35305] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs 2023-02-13 12:00:20,145 INFO [RS:0;jenkins-hbase16:35305] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C35305%2C1676289577427:(num 1676289579117) 2023-02-13 12:00:20,145 DEBUG [RS:0;jenkins-hbase16:35305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:20,145 INFO [RS:0;jenkins-hbase16:35305] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:00:20,146 INFO [RS:0;jenkins-hbase16:35305] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 12:00:20,147 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:00:20,147 INFO [RS:0;jenkins-hbase16:35305] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:35305 2023-02-13 12:00:20,148 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,45135,1676289577478 already deleted, retry=false 2023-02-13 12:00:20,148 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,45135,1676289577478 expired; onlineServers=2 2023-02-13 12:00:20,157 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:20,157 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,35305,1676289577427 2023-02-13 12:00:20,157 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:00:20,157 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3147eceb rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,165 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,35305,1676289577427] 2023-02-13 12:00:20,165 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,35305,1676289577427; numProcessing=2 2023-02-13 12:00:20,173 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,35305,1676289577427 already deleted, retry=false 2023-02-13 12:00:20,173 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,35305,1676289577427 expired; onlineServers=1 2023-02-13 12:00:20,241 DEBUG [RS:2;jenkins-hbase16:34233] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/oldWALs 2023-02-13 12:00:20,241 INFO [RS:2;jenkins-hbase16:34233] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C34233%2C1676289577518:(num 1676289579117) 2023-02-13 12:00:20,242 DEBUG [RS:2;jenkins-hbase16:34233] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:20,242 INFO [RS:2;jenkins-hbase16:34233] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:00:20,242 INFO [RS:2;jenkins-hbase16:34233] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 12:00:20,242 INFO [RS:2;jenkins-hbase16:34233] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 12:00:20,242 INFO [RS:2;jenkins-hbase16:34233] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 12:00:20,242 INFO [RS:2;jenkins-hbase16:34233] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 12:00:20,242 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:00:20,243 INFO [RS:2;jenkins-hbase16:34233] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:34233 2023-02-13 12:00:20,252 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34233,1676289577518 2023-02-13 12:00:20,252 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:00:20,252 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6cbe9d65 rejected from java.util.concurrent.ThreadPoolExecutor@6881042a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,260 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,34233,1676289577518] 2023-02-13 12:00:20,260 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,34233,1676289577518; numProcessing=3 2023-02-13 12:00:20,268 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,34233,1676289577518 already deleted, retry=false 2023-02-13 12:00:20,268 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,34233,1676289577518 expired; onlineServers=0 2023-02-13 12:00:20,269 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,39205,1676289576353' ***** 2023-02-13 12:00:20,269 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 12:00:20,270 DEBUG [M:0;jenkins-hbase16:39205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e08b828, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 12:00:20,271 INFO [M:0;jenkins-hbase16:39205] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,39205,1676289576353 2023-02-13 12:00:20,271 INFO [M:0;jenkins-hbase16:39205] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,39205,1676289576353; all regions closed. 2023-02-13 12:00:20,271 DEBUG [M:0;jenkins-hbase16:39205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:00:20,271 DEBUG [M:0;jenkins-hbase16:39205] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 12:00:20,271 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 12:00:20,272 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1676289578668] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1676289578668,5,FailOnTimeoutGroup] 2023-02-13 12:00:20,272 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1676289578667] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1676289578667,5,FailOnTimeoutGroup] 2023-02-13 12:00:20,272 DEBUG [M:0;jenkins-hbase16:39205] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 12:00:20,273 INFO [M:0;jenkins-hbase16:39205] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 12:00:20,273 INFO [M:0;jenkins-hbase16:39205] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 12:00:20,274 INFO [M:0;jenkins-hbase16:39205] hbase.ChoreService(369): Chore service for: master/jenkins-hbase16:0 had [] on shutdown 2023-02-13 12:00:20,274 DEBUG [M:0;jenkins-hbase16:39205] master.HMaster(1502): Stopping service threads 2023-02-13 12:00:20,274 INFO [M:0;jenkins-hbase16:39205] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 12:00:20,275 ERROR [M:0;jenkins-hbase16:39205] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] 2023-02-13 12:00:20,276 INFO [M:0;jenkins-hbase16:39205] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 12:00:20,277 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 12:00:20,282 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 12:00:20,282 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:00:20,282 DEBUG [M:0;jenkins-hbase16:39205] zookeeper.ZKUtil(398): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 12:00:20,282 WARN [M:0;jenkins-hbase16:39205] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 12:00:20,282 INFO [M:0;jenkins-hbase16:39205] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 12:00:20,282 INFO [M:0;jenkins-hbase16:39205] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 12:00:20,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 12:00:20,282 DEBUG [M:0;jenkins-hbase16:39205] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 12:00:20,282 INFO [M:0;jenkins-hbase16:39205] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:00:20,282 DEBUG [M:0;jenkins-hbase16:39205] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:00:20,282 DEBUG [M:0;jenkins-hbase16:39205] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 12:00:20,282 DEBUG [M:0;jenkins-hbase16:39205] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:00:20,282 INFO [M:0;jenkins-hbase16:39205] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=844.40 KB heapSize=1.01 MB 2023-02-13 12:00:20,298 INFO [M:0;jenkins-hbase16:39205] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=844.40 KB at sequenceid=2048 (bloomFilter=true), to=hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8fe77c4b1e3f43f7971bdd7e66ea377c 2023-02-13 12:00:20,300 INFO [M:0;jenkins-hbase16:39205] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8fe77c4b1e3f43f7971bdd7e66ea377c 2023-02-13 12:00:20,301 DEBUG [M:0;jenkins-hbase16:39205] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8fe77c4b1e3f43f7971bdd7e66ea377c as hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8fe77c4b1e3f43f7971bdd7e66ea377c 2023-02-13 12:00:20,303 INFO [M:0;jenkins-hbase16:39205] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8fe77c4b1e3f43f7971bdd7e66ea377c 2023-02-13 12:00:20,303 INFO [M:0;jenkins-hbase16:39205] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41081/user/jenkins/test-data/693b6811-305f-8725-3aec-a3d8037bdef2/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8fe77c4b1e3f43f7971bdd7e66ea377c, entries=282, sequenceid=2048, filesize=30.7 K 2023-02-13 12:00:20,304 INFO [M:0;jenkins-hbase16:39205] regionserver.HRegion(2947): Finished flush of dataSize ~844.40 KB/864666, heapSize ~1.01 MB/1056384, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=2048, compaction requested=false 2023-02-13 12:00:20,304 INFO [M:0;jenkins-hbase16:39205] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:00:20,304 DEBUG [M:0;jenkins-hbase16:39205] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676289620282Disabling compacts and flushes for region at 1676289620282Disabling writes for close at 1676289620282Obtaining lock to block concurrent updates at 1676289620282Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676289620282Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=864666, getHeapSize=1056384, getOffHeapSize=0, getCellsCount=2535 at 1676289620283 (+1 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676289620283Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676289620283Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676289620289 (+6 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676289620289Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676289620301 (+12 ms)Finished flush of dataSize ~844.40 KB/864666, heapSize ~1.01 MB/1056384, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=2048, compaction requested=false at 1676289620304 (+3 ms)Writing region close event to WAL at 1676289620304Closed at 1676289620304 2023-02-13 12:00:20,308 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:00:20,308 INFO [M:0;jenkins-hbase16:39205] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 12:00:20,308 INFO [M:0;jenkins-hbase16:39205] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:39205 2023-02-13 12:00:20,313 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,313 INFO [RS:0;jenkins-hbase16:35305] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,35305,1676289577427; zookeeper connection closed. 2023-02-13 12:00:20,313 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f6e1ee9 rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,313 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:35305-0x10052c279140001, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,313 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4e67ae40] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4e67ae40 2023-02-13 12:00:20,313 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@37be35de rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,318 DEBUG [M:0;jenkins-hbase16:39205] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase16.apache.org,39205,1676289576353 already deleted, retry=false 2023-02-13 12:00:20,413 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,413 INFO [RS:1;jenkins-hbase16:45135] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,45135,1676289577478; zookeeper connection closed. 2023-02-13 12:00:20,413 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d8224fc rejected from java.util.concurrent.ThreadPoolExecutor@19bda394[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,414 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2c279bd5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2c279bd5 2023-02-13 12:00:20,414 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:45135-0x10052c279140002, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,414 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6100cf5c rejected from java.util.concurrent.ThreadPoolExecutor@19bda394[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,513 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,513 INFO [M:0;jenkins-hbase16:39205] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,39205,1676289576353; zookeeper connection closed. 2023-02-13 12:00:20,514 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3448ce5c rejected from java.util.concurrent.ThreadPoolExecutor@5dcc3055[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,514 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): master:39205-0x10052c279140000, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,514 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@872b40a rejected from java.util.concurrent.ThreadPoolExecutor@5dcc3055[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,614 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,614 INFO [RS:2;jenkins-hbase16:34233] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,34233,1676289577518; zookeeper connection closed. 2023-02-13 12:00:20,614 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4293e0b rejected from java.util.concurrent.ThreadPoolExecutor@6881042a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,614 DEBUG [Listener at localhost.localdomain/36409-EventThread] zookeeper.ZKWatcher(600): regionserver:34233-0x10052c279140003, quorum=127.0.0.1:63172, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:00:20,614 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@70c0349a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@70c0349a 2023-02-13 12:00:20,614 ERROR [Listener at localhost.localdomain/36409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7c2429ac rejected from java.util.concurrent.ThreadPoolExecutor@6881042a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:00:20,615 INFO [Listener at localhost.localdomain/36409] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 12:00:20,616 WARN [Listener at localhost.localdomain/36409] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 12:00:20,622 INFO [Listener at localhost.localdomain/36409] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 12:00:20,736 WARN [BP-1187519313-188.40.62.62-1676289572673 heartbeating to localhost.localdomain/127.0.0.1:41081] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 12:00:20,736 WARN [BP-1187519313-188.40.62.62-1676289572673 heartbeating to localhost.localdomain/127.0.0.1:41081] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1187519313-188.40.62.62-1676289572673 (Datanode Uuid 14795f5c-1d04-4a18-8e7a-59414b03ca82) service to localhost.localdomain/127.0.0.1:41081 2023-02-13 12:00:20,738 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data5/current/BP-1187519313-188.40.62.62-1676289572673] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:00:20,738 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data6/current/BP-1187519313-188.40.62.62-1676289572673] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:00:20,741 WARN [Listener at localhost.localdomain/36409] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 12:00:20,743 INFO [Listener at localhost.localdomain/36409] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 12:00:20,850 WARN [BP-1187519313-188.40.62.62-1676289572673 heartbeating to localhost.localdomain/127.0.0.1:41081] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 12:00:20,850 WARN [BP-1187519313-188.40.62.62-1676289572673 heartbeating to localhost.localdomain/127.0.0.1:41081] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1187519313-188.40.62.62-1676289572673 (Datanode Uuid e81e25ff-5c5b-437b-b4c3-1df197948ce3) service to localhost.localdomain/127.0.0.1:41081 2023-02-13 12:00:20,851 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data3/current/BP-1187519313-188.40.62.62-1676289572673] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:00:20,852 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data4/current/BP-1187519313-188.40.62.62-1676289572673] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:00:20,854 WARN [Listener at localhost.localdomain/36409] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 12:00:20,856 INFO [Listener at localhost.localdomain/36409] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 12:00:20,966 WARN [BP-1187519313-188.40.62.62-1676289572673 heartbeating to localhost.localdomain/127.0.0.1:41081] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 12:00:20,966 WARN [BP-1187519313-188.40.62.62-1676289572673 heartbeating to localhost.localdomain/127.0.0.1:41081] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1187519313-188.40.62.62-1676289572673 (Datanode Uuid 27381562-f408-4782-a1fa-cd8daf0674dd) service to localhost.localdomain/127.0.0.1:41081 2023-02-13 12:00:20,968 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data1/current/BP-1187519313-188.40.62.62-1676289572673] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:00:20,968 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0eb4737e-96a8-9632-5e11-9efa90b66165/cluster_80b5f972-181b-9796-2403-8a07d8ff7c23/dfs/data/data2/current/BP-1187519313-188.40.62.62-1676289572673] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:00:20,992 INFO [Listener at localhost.localdomain/36409] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-02-13 12:00:21,117 INFO [Listener at localhost.localdomain/36409] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 12:00:21,190 INFO [Listener at localhost.localdomain/36409] hbase.HBaseTestingUtility(1293): Minicluster is down